{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://i.imgur.com/eBRPvWB.png)\n",
    "\n",
    "# 使用seq2seq网络和注意力机制实现汉-英翻译\n",
    "\n",
    "这个项目，我们会实现一个汉语到英语的神经网络翻译模型。\n",
    "\n",
    "\n",
    "这里实现的是论文[seq2seq 网络](http://arxiv.org/abs/1409.3215)的系统，它通过两个RNN来实现把一个训练翻译成两位一个序列。Encoder把输入序列编码成一个向量，而Decoder会把这个向量解码成两位一个输出序列。\n",
    "\n",
    "为了提升效果，我们使用了[注意力机制](https://arxiv.org/abs/1409.0473)，它可以让decoder学习到在翻译的过程中怎么重点关注输入序列的某一部分。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## sequence to sequence模型\n",
    "\n",
    "sequence to sequence模型，或者说seq2seq模型，由两个RNN组成。这两个RNN分别叫做encoder和decoder。\n",
    "encoder会一个词一个出的读入输入序列，每一步都有一个输出，而最后的输出叫做context向量，我们可以认为是模型对源语言句子“语义”的一种表示。而decoder用这个context向量一步一步的生成目标语言的句子。\n",
    "\n",
    "![](seq2seq.png)\n",
    "\n",
    "为什么要两个RNN呢，如果我们使用一个RNN，输入和输出是一对一的关系（对于分类，我们可以只使用最后一个输出），但是翻译肯定不是一个词对一个词的翻译。当然这只是使用两个RNN在形式上的方便，从“原理”上来说，人类翻译也是类似的，首先仔细阅读源语句，然后“理解”它，而所谓的“理解”在seq2seq模型里可以认为encoding的过程，然后再根据理解，翻译成目标语句。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 注意力机制\n",
    "\n",
    "用一个固定长度的向量来承载输入序列的完整“语义”，不管向量能有多长，都是很困难的事情。\n",
    "\n",
    "[Bahdanau et al.等人引入的](https://arxiv.org/abs/1409.0473) **注意力机制**试图这样来解决这个问题：我们不依赖于一个固定长度的向量，而是通过“注意”输入的某些部分。在decoer每一步解码的时候都通过这个机制来选择输入的一部分来重点考虑。这似乎是合乎人类的翻译过程的——我们首先通过一个encoder大致理解句子的意思（编码到一个定长向量），具体翻译某个词或者短语的时候我们会仔细推敲对应的源语言的词（注意力机制）。\n",
    "\n",
    "![](5y6SCvU.png)\n",
    "\n",
    "注意力是通过decoder的另外一个神经网络层来计算的。它的输入是当前输入和上一个时刻的隐状态，输出是一个新的向量，这个向量的长度和输入相同（因为输入是变长的，我们会选择一个最大的长度），这个向量会用softmax变成“概率”，得到*注意力权重*，这个权重可以认为需要花费多大的“注意力”到某个输入上，因此我们会用这个权重加权平均encoder的输出，从而得到一个新的context向量，这个向量会用来预测当前时刻的输出。\n",
    "\n",
    "![](K1qMPxs.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 依赖\n",
    "\n",
    "我们需要[PyTorch](http://pytorch.org/)来构建和训练模型，需要[matplotlib](https://matplotlib.org/) 来绘图，剩下的都是Python自带的库。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import unicodedata\n",
    "import string\n",
    "import re\n",
    "import random\n",
    "import time\n",
    "import datetime\n",
    "import math\n",
    "import socket\n",
    "hostname = socket.gethostname()\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n",
    "from torch.nn.utils.rnn import pad_packed_sequence, pack_padded_sequence#, masked_cross_entropy\n",
    "from masked_cross_entropy import *\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams[u'font.sans-serif'] = ['simhei']\n",
    "plt.rcParams['axes.unicode_minus'] = False\n",
    "import matplotlib.ticker as ticker\n",
    "import numpy as np\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果没有GPU，请设置为False，但是训练时间可能会比较长。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "USE_CUDA = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 加载数据\n",
    "\n",
    "训练数据是几千个英语到汉语的平行句对。我们这里只是介绍算法，所以使用一个很小的数据集来演示。数据在data/eng-chn.txt里，格式如下：\n",
    "```\n",
    "I won!  我 赢 了 。\n",
    "```\n",
    "\n",
    "每一行都是一个句对，用tab分割，中文部分已经提前分完词了(空格分开，所以split就可以得到中文的词)。\n",
    "\n",
    "我们会用one-hot的方法来表示一个单词。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 单词变成数字\n",
    "\n",
    "我们会创建一个Lang对象来表示源/目标语言，它包含word2idx、idx2word和word2count，分别表示单词到id、id到单词和单词的词频。\n",
    "word2count的作用是用于过滤一些低频词（把它变成unknown）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#按字分词，目前不需要了。\n",
    "def seg_cn_words(s):\n",
    "    words=[]\n",
    "    for ch in s:\n",
    "        if ch !=' ':\n",
    "            words.append(ch)\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "PAD_token = 0\n",
    "SOS_token = 1\n",
    "EOS_token = 2\n",
    "\n",
    "class Lang:\n",
    "    def __init__(self, name):\n",
    "        self.name = name\n",
    "        self.trimmed = False\n",
    "        self.word2index = {}\n",
    "        self.word2count = {}\n",
    "        self.index2word = {0: \"PAD\", 1: \"SOS\", 2: \"EOS\"}\n",
    "        self.n_words = 3 # Count default tokens\n",
    "\n",
    "    def seg_words(self, sentence):\n",
    "#        if self.name=='chn':\n",
    "#            return seg_cn_words(sentence)\n",
    "        return sentence.split(' ')\n",
    "        \n",
    "    def index_words(self, sentence):\n",
    "        words=self.seg_words(sentence)\n",
    "        for word in words:\n",
    "            self.index_word(word)\n",
    "\n",
    "    def index_word(self, word):\n",
    "        if word not in self.word2index:\n",
    "            self.word2index[word] = self.n_words\n",
    "            self.word2count[word] = 1\n",
    "            self.index2word[self.n_words] = word\n",
    "            self.n_words += 1\n",
    "        else:\n",
    "            self.word2count[word] += 1\n",
    "\n",
    "    # 删除掉频率少于阈值的此。\n",
    "    def trim(self, min_count):\n",
    "        if self.trimmed: return\n",
    "        self.trimmed = True\n",
    "        \n",
    "        keep_words = []\n",
    "        \n",
    "        for k, v in self.word2count.items():\n",
    "            if v >= min_count:\n",
    "                keep_words.append(k)\n",
    "\n",
    "        print('keep_words %s / %s = %.4f' % (\n",
    "            len(keep_words), len(self.word2index), len(keep_words) / len(self.word2index)\n",
    "        ))\n",
    "\n",
    "        # 重新构建dict\n",
    "        self.word2index = {}\n",
    "        self.word2count = {}\n",
    "        self.index2word = {0: \"PAD\", 1: \"SOS\", 2: \"EOS\"}\n",
    "        self.n_words = 3 # 默认3个词。\n",
    "\n",
    "        for word in keep_words:\n",
    "            self.index_word(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 读取和预处理 \n",
    "\n",
    "我们会只保留文本里的英文和中文字符，把大小转换成小写。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def unicode_to_ascii(s):\n",
    "    return ''.join(\n",
    "        c for c in unicodedata.normalize('NFD', s)\n",
    "        if unicodedata.category(c) != 'Mn'\n",
    "    )\n",
    "\n",
    "# 大写转小写，trim，移除常见标点，英文和中文之外的字符。\n",
    "def normalize_string(s):\n",
    "    s = unicode_to_ascii(s.lower().strip())\n",
    "    s = re.sub(r\"([,.!?])\", r\" \\1 \", s)\n",
    "    s = re.sub(r\"[^a-zA-Z\\u4e00-\\u9fa5,.!?，。！？]+\", r\" \", s)\n",
    "    s = re.sub(r\"\\s+\", r\" \", s).strip()\n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "中文word  cn!\n",
      "中文word cn !\n"
     ]
    }
   ],
   "source": [
    "s=unicode_to_ascii(\"中文word  cn!\")\n",
    "print(s)\n",
    "s=normalize_string(s)\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "文件格式是每行一个句对，用tab分隔，第一个是英语，第二个是汉语，为了方便未来的复用，我们有一个reverse参数，这样如果我们需要汉语到英语的翻译就可以用到。如果读者想实现英语到汉英的翻译系统，那么只需要reverse为False就行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_langs(lang1, lang2, reverse=False):\n",
    "    print(\"Reading lines...\")\n",
    "\n",
    "    # 读取文件\n",
    "    filename = '../data/%s-%s.txt' % (lang1, lang2)\n",
    "    lines = open(filename).read().strip().split('\\n')\n",
    "\n",
    "    # Split\n",
    "    pairs = [[normalize_string(s) for s in l.split('\\t')] for l in lines]\n",
    "\n",
    "    # 反转句对，从英汉变成汉英\n",
    "    if reverse:\n",
    "        pairs = [list(reversed(p)) for p in pairs]\n",
    "        input_lang = Lang(lang2)\n",
    "        output_lang = Lang(lang1)\n",
    "    else:\n",
    "        input_lang = Lang(lang1)\n",
    "        output_lang = Lang(lang2)\n",
    "\n",
    "    return input_lang, output_lang, pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "MIN_LENGTH = 3\n",
    "MAX_LENGTH = 25\n",
    "\n",
    "def filter_pairs(pairs):\n",
    "    filtered_pairs = []\n",
    "    for pair in pairs:\n",
    "        if len(pair[0]) >= MIN_LENGTH and len(pair[0]) <= MAX_LENGTH \\\n",
    "            and len(pair[1]) >= MIN_LENGTH and len(pair[1]) <= MAX_LENGTH:\n",
    "                filtered_pairs.append(pair)\n",
    "    return filtered_pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据处理过程如下：\n",
    "\n",
    "* 读取文件，split成行，再split成pair\n",
    "* 文本归一化，过滤掉过短(小于3)和过长(大于25)的句对。\n",
    "* 通过pair里的句子得到单词列表\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading lines...\n",
      "Read 19777 sentence pairs\n",
      "Filtered to 5095 pairs\n",
      "Indexing words...\n",
      "Indexed 2930 words in input language, 2258 words in output\n"
     ]
    }
   ],
   "source": [
    "def prepare_data(lang1_name, lang2_name, reverse=False):\n",
    "    input_lang, output_lang, pairs = read_langs(lang1_name, lang2_name, reverse)\n",
    "    print(\"Read %d sentence pairs\" % len(pairs))\n",
    "    \n",
    "    pairs = filter_pairs(pairs)\n",
    "    print(\"Filtered to %d pairs\" % len(pairs))\n",
    "    \n",
    "    print(\"Indexing words...\")\n",
    "    for pair in pairs:\n",
    "        input_lang.index_words(pair[0])\n",
    "        output_lang.index_words(pair[1])\n",
    "    \n",
    "    print('Indexed %d words in input language, %d words in output' % (input_lang.n_words, output_lang.n_words))\n",
    "    return input_lang, output_lang, pairs\n",
    "\n",
    "input_lang, output_lang, pairs = prepare_data('eng', 'chn', True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 过滤词汇 \n",
    "\n",
    "我们会取得词频低于5的词语。对于低频的词和词典之外的词(OOV)，系统很难学习，这是学术研究的一个难点，我们暂时忽略掉这些低频词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "keep_words 578 / 2927 = 0.1975\n",
      "keep_words 568 / 2255 = 0.2519\n"
     ]
    }
   ],
   "source": [
    "MIN_COUNT = 5\n",
    "\n",
    "input_lang.trim(MIN_COUNT)\n",
    "output_lang.trim(MIN_COUNT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chn\n"
     ]
    }
   ],
   "source": [
    "print(input_lang.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 过滤句对\n",
    "\n",
    "我们接下来去掉包含低频词或者OOV的句对。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trimmed from 5095 pairs to 1854, 0.3639 of total\n"
     ]
    }
   ],
   "source": [
    "keep_pairs = []\n",
    "\n",
    "for pair in pairs:\n",
    "    input_sentence = pair[0]\n",
    "    output_sentence = pair[1]\n",
    "    keep_input = True\n",
    "    keep_output = True\n",
    "    words=input_lang.seg_words(input_sentence)\n",
    "    for word in words:\n",
    "        if word not in input_lang.word2index:\n",
    "            keep_input = False\n",
    "            break\n",
    "            \n",
    "    words=output_lang.seg_words(output_sentence)\n",
    "    for word in words:\n",
    "        if word not in output_lang.word2index:\n",
    "            keep_output = False\n",
    "            break\n",
    "\n",
    "    # Remove if pair doesn't match input and output conditions\n",
    "    if keep_input and keep_output:\n",
    "        keep_pairs.append(pair)\n",
    "\n",
    "print(\"Trimmed from %d pairs to %d, %.4f of total\" % (len(pairs), len(keep_pairs), len(keep_pairs) / len(pairs)))\n",
    "pairs = keep_pairs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['我 赢 了 。', 'i won !']\n"
     ]
    }
   ],
   "source": [
    "print(pairs[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 把训练数据变成Tensor/Variable\n",
    "\n",
    "为了让神经网络能够处理，我们首先需要把句子变成Tensor。每个句子首先分成词，每个词被替换成对应的index。另外我们会增加一个特殊的EOS来表示句子的结束。\n",
    "\n",
    "![](https://i.imgur.com/LzocpGH.png)\n",
    "\n",
    "在PyTorch里一个Tensor是一个多维数组，它的所有元素的数据类型都是一样的。我们这里使用LongTensor来表示词的index。\n",
    "\n",
    "可以训练的PyTorch模块要求输入是Variable而不是Tensor。变量除了包含Tensor的内容之外，它还会跟踪计算图的状态，从而可以进行自动梯度的求值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 把句子变成index的list，最后加上EOS\n",
    "def indexes_from_sentence(lang, sentence):\n",
    "    idxs=[]\n",
    "    words=lang.seg_words(sentence)\n",
    "    for word in words:\n",
    "        if word in lang.word2index:\n",
    "            idxs.append(lang.word2index[word])\n",
    "    idxs.append(EOS_token)\n",
    "    return idxs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了更好的使用GPU，我们一次训练一个batch的数据，但是问题是不同句对的长度是不一样的。解决办法是通过“pad”来把短的序列补到和长的序列一样长，我们需要一个特殊的整数来表示它是一个pad值而不是其他的词，我们这里使用0。当计算loss的时候，我们也需要把这些0对应的loss去掉（因为实际的序列到这里已经没有了，但是padding之后任何会计算出模型的预测值，从而有loss）\n",
    "\n",
    "\n",
    "![](https://i.imgur.com/gGlkEEF.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 把一个序列padding到长度为max_length。\n",
    "def pad_seq(seq, max_length):\n",
    "    seq += [PAD_token for i in range(max_length - len(seq))]\n",
    "    return seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练一次需要一个batch的数据，我们一般随机的抽取bach_size个句对，然后把它们padding到最长的序列长度。此外，我们需要记录下未padding之前的长度，因为计算loss的时候会需要用到它。\n",
    "\n",
    "我们会用list的list来初始化LongTensor，这个list的大小是batch_size，list的每个元素是一个序列(句子)。这样我们可以得到一个`(batch_size x max_len)`的Tensor，但是因为训练时我们需要逐个时刻的计算batch_size个数据，所以我们需要把它转置成`(max_len x batch_size)`。这样tensor[t]就表示t时刻的batch_size个词。\n",
    "\n",
    "\n",
    "![](https://i.imgur.com/nBxTG3v.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_batch(batch_size):\n",
    "    input_seqs = []\n",
    "    target_seqs = []\n",
    "\n",
    "    # 随机选择pairs\n",
    "    for i in range(batch_size):\n",
    "        pair = random.choice(pairs)\n",
    "        input_seqs.append(indexes_from_sentence(input_lang, pair[0]))\n",
    "        target_seqs.append(indexes_from_sentence(output_lang, pair[1]))\n",
    "\n",
    "    # 把输入和输出序列zip起来，通过输入的长度降序排列，然后unzip\n",
    "    seq_pairs = sorted(zip(input_seqs, target_seqs), key=lambda p: len(p[0]), reverse=True)\n",
    "    input_seqs, target_seqs = zip(*seq_pairs)\n",
    "    \n",
    "    # 对输入和输出序列都进行padding。\n",
    "    input_lengths = [len(s) for s in input_seqs]\n",
    "    input_padded = [pad_seq(s, max(input_lengths)) for s in input_seqs]\n",
    "    target_lengths = [len(s) for s in target_seqs]\n",
    "    target_padded = [pad_seq(s, max(target_lengths)) for s in target_seqs]\n",
    "\n",
    "    # padding之后的shape是(batch_size x max_len)，我们需要把它转置成(max_len x batch_size)\n",
    "    input_var = Variable(torch.LongTensor(input_padded)).transpose(0, 1)\n",
    "    target_var = Variable(torch.LongTensor(target_padded)).transpose(0, 1)\n",
    "    \n",
    "    if USE_CUDA:\n",
    "        input_var = input_var.cuda()\n",
    "        target_var = target_var.cuda()\n",
    "        \n",
    "    return input_var, input_lengths, target_var, target_lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以测试一下，它会返回两个`(max_len x batch_size)` tensor，这是输入和输出序列，同时也会返回padding之前的实际长度。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[ 11,  11],\n",
       "         [258, 147],\n",
       "         [147, 144],\n",
       "         [225,   3],\n",
       "         [  3,   2],\n",
       "         [  2,   0]], device='cuda:0'), [6, 5], tensor([[  7,   7],\n",
       "         [280, 149],\n",
       "         [ 93, 274],\n",
       "         [149, 146],\n",
       "         [ 22,   3],\n",
       "         [  3,   2],\n",
       "         [  2,   0]], device='cuda:0'), [7, 6])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_batch(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 构建模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoder\n",
    "\n",
    "<img src=\"images/encoder-network.png\" style=\"float: right\" />\n",
    "\n",
    "\n",
    "encoder的输入是batch大小的词序列——大小是`(max_len x batch_size)``LongTensor`，每个词都会输出一个向量，所以最终的输出是大小为  `(max_len x batch_size x hidden_size)`的`FloatTensor`。\n",
    "\n",
    "输入序列的词会被fed到一个[embedding层 `nn.Embedding`](http://pytorch.org/docs/nn.html#embedding)从而得到每个词的embedding，这样得到一个`(max_len x batch_size x embedding_size【这里embedding_size==hidden_size，但这不是必须相等的】)`的Tensor，这个Tensor会输入到[GRU层 `nn.GRU`](http://pytorch.org/docs/nn.html#gru)。最终GRU的输出是`seq_len x batch_size x hidden_size`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderRNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, n_layers=1, dropout=0.1):\n",
    "        super(EncoderRNN, self).__init__()\n",
    "        \n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.n_layers = n_layers\n",
    "        self.dropout = dropout\n",
    "        \n",
    "        self.embedding = nn.Embedding(input_size, hidden_size)\n",
    "        self.gru = nn.GRU(hidden_size, hidden_size, n_layers, dropout=self.dropout, bidirectional=True)\n",
    "        \n",
    "    def forward(self, input_seqs, input_lengths, hidden=None):\n",
    "        # 注意：和前面的实现不同，这里没有时刻t的for循环，而是一次输入GRU直接计算出来\n",
    "        embedded = self.embedding(input_seqs)\n",
    "        packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, input_lengths)\n",
    "        outputs, hidden = self.gru(packed, hidden)\n",
    "        outputs, output_lengths = torch.nn.utils.rnn.pad_packed_sequence(outputs) # unpack\n",
    "        hidden = self._cat_directions(hidden)\n",
    "        return outputs, hidden\n",
    "    \n",
    "    def _cat_directions(self, hidden):\n",
    "        \"\"\" If the encoder is bidirectional, do the following transformation.\n",
    "            Ref: https://github.com/IBM/pytorch-seq2seq/blob/master/seq2seq/models/DecoderRNN.py#L176\n",
    "            -----------------------------------------------------------\n",
    "            In: (num_layers * num_directions, batch_size, hidden_size)\n",
    "            (ex: num_layers=2, num_directions=2)\n",
    "\n",
    "            layer 1: forward__hidden(1)\n",
    "            layer 1: backward_hidden(1)\n",
    "            layer 2: forward__hidden(2)\n",
    "            layer 2: backward_hidden(2)\n",
    "\n",
    "            -----------------------------------------------------------\n",
    "            Out: (num_layers, batch_size, hidden_size * num_directions)\n",
    "\n",
    "            layer 1: forward__hidden(1) backward_hidden(1)\n",
    "            layer 2: forward__hidden(2) backward_hidden(2)\n",
    "        \"\"\"\n",
    "        def _cat(h):\n",
    "            return torch.cat([h[0:h.size(0):2], h[1:h.size(0):2]], 2)\n",
    "            \n",
    "        if isinstance(hidden, tuple):\n",
    "            # LSTM hidden contains a tuple (hidden state, cell state)\n",
    "            hidden = tuple([_cat(h) for h in hidden])\n",
    "        else:\n",
    "            # GRU hidden\n",
    "            hidden = _cat(hidden)\n",
    "            \n",
    "        return hidden    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention Decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 理解Bahdanau等人提出的模型\n",
    "\n",
    "下面我们来学习一下这篇文章提出的 [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473) Attention Decoder。\n",
    "\n",
    "decoder在每一个时刻的输出依赖与前一个时刻的输出和一个$\\mathbf x$，这个$\\mathbf x$包括当前的隐状态（它也会考虑前一个时刻的输出）和一个注意力”context“，下文会介绍它。函数$g$是一个带非线性激活的全连接层，它的输入是$y_{i-1}$, $s_i$ 和 $c_i$拼接起来的。\n",
    "\n",
    "$$\n",
    "p(y_i \\mid \\{y_1,...,y_{i-1}\\},\\mathbf{x}) = g(y_{i-1}, s_i, c_i)\n",
    "$$\n",
    "\n",
    "上式的意思是：我们在翻译当前词时只考虑上一个翻译出来的词以及当前的隐状态和注意力context。\n",
    "\n",
    "当前隐状态$s_i$是有RNN $f$计算出来的，这个RNN的输入是上一个隐状态$s_{i-1}$，decoder的上一个输出$y_{i-1}$和 context向量$c_i$。\n",
    "\n",
    "在代码实现中，我们使用的RNN是`nn.GRU`，隐状态 $s_i$是`hidden`，输出$y_i$是`output`， context $c_i$ 是`context`。\n",
    "\n",
    "$$\n",
    "s_i = f(s_{i-1}, y_{i-1}, c_i)\n",
    "$$\n",
    "\n",
    "context向量$c_i$是encoder在每个时刻(词)的输出的加权和，而权值$a_{ij}$表示i时刻需要关注$h_j$的程度(概率)。\n",
    "\n",
    "$$\n",
    "c_i = \\sum_{j=1}^{T_x} a_{ij} h_j\n",
    "$$\n",
    "\n",
    "而权值$a_{ij}$是\"能量\" $e_{ij}$的softmax。\n",
    "\n",
    "$$\n",
    "a_{ij} = \\dfrac{exp(e_{ij})}{\\sum_{k=1}^{T} exp(e_{ik})}\n",
    "$$\n",
    "\n",
    "而能量$e_{ij}$是上个时刻的隐状态$s_{i-1}$和encoder第j个时刻的输出$h_j$的函数：\n",
    "\n",
    "$$\n",
    "e_{ij} = a(s_{i-1}, h_j)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Luong等人提出的模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luong等人在[Effective Approaches to Attention-based Neural Machine Translation](https://arxiv.org/abs/1508.04025)提出了更多的提高和简化。他们描述了”全局注意力“模型，其计算注意力得分的方法和之前不同。前面是通过$s_{i-1}$和$h_j$计算$a_{ij}$，也就是当前的注意力权重依赖与前一个状态，而这里的注意力依赖与decoder当前的隐状态和encoder所有隐状态：\n",
    "\n",
    "$$\n",
    "a_t(s) = align(h_t, \\bar h_s)  = \\dfrac{exp(score(h_t, \\bar h_s))}{\\sum_{s'} exp(score(h_t, \\bar h_{s'}))}\n",
    "$$\n",
    "\n",
    "特点的\"score\"函数会比较两个隐状态的”相似度“，可以是两个向量的内积，也可以是$h_{s'}$做一个线性变换之后和$h_t$的内积，也可以是把两个向量拼接起来然后做一个线性变换，然后和一个参数$v_a$（这个参数是学习出来的）的内积：\n",
    "\n",
    "$$\n",
    "score(h_t, \\bar h_s) =\n",
    "\\begin{cases}\n",
    "h_t ^\\top \\bar h_s & dot \\\\\n",
    "h_t ^\\top \\textbf{W}_a \\bar h_s & general \\\\\n",
    "v_a ^\\top \\textbf{W}_a [ h_t ; \\bar h_s ] & concat\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "scoring函数的模块化定义使得我们可以随意的修改而不影响其它地方的代码。这个模块的输入总是decoder的隐状态和encoder的所有输出。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 计算注意力的通用模块\n",
    "\n",
    "为了代码重用，这里把注意力的计算封装成一个模块，不管是哪个算法都可以复用它。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Attn(nn.Module):\n",
    "    def __init__(self, method, hidden_size):\n",
    "        super(Attn, self).__init__()\n",
    "        \n",
    "        self.method = method\n",
    "        self.hidden_size = hidden_size\n",
    "        \n",
    "        if self.method == 'general':\n",
    "            self.attn = nn.Linear(self.hidden_size*2, hidden_size*2)\n",
    "\n",
    "        elif self.method == 'concat':\n",
    "            self.attn = nn.Linear(self.hidden_size * 2, hidden_size)\n",
    "            self.v = nn.Parameter(torch.FloatTensor(1, hidden_size))\n",
    "\n",
    "    def forward(self, hidden, encoder_outputs):\n",
    "        max_len = encoder_outputs.size(0)\n",
    "        this_batch_size = encoder_outputs.size(1)\n",
    "\n",
    "        # 创建变量来存储注意力能量\n",
    "        attn_energies = Variable(torch.zeros(this_batch_size, max_len)) # B x S\n",
    "\n",
    "        if USE_CUDA:\n",
    "            attn_energies = attn_energies.cuda()\n",
    "\n",
    "        # 计算\n",
    "        for b in range(this_batch_size):\n",
    "            for i in range(max_len):\n",
    "                attn_energies[b, i] = self.score(hidden[:, b], encoder_outputs[i, b].unsqueeze(0))\n",
    "\n",
    "        # softmax并且resize成1 x B x S\n",
    "        return F.softmax(attn_energies, dim=1).unsqueeze(1)\n",
    "    \n",
    "        \n",
    "    def score(self, hidden, encoder_output):\n",
    "\n",
    "        if self.method == 'dot':\n",
    "            energy = hidden.view(-1).dot(encoder_output.view(-1))\n",
    "            return energy\n",
    "\n",
    "        elif self.method == 'general':\n",
    "            energy = self.attn(encoder_output)\n",
    "            energy = hidden.view(-1).dot(energy.view(-1))\n",
    "            return energy\n",
    "\n",
    "        elif self.method == 'concat':\n",
    "            energy = self.attn(torch.cat((hidden, encoder_output), 1))\n",
    "            energy = self.other.view(-1).dot(energy.view(-1))\n",
    "            return energy        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们可以构建一个decoder，它会把Attn模块放到RNN之后用来计算注意力的权重，并且用它来计算context向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LuongAttnDecoderRNN(nn.Module):\n",
    "    def __init__(self, attn_model, hidden_size, output_size, n_layers=1, dropout=0.1):\n",
    "        super(LuongAttnDecoderRNN, self).__init__()\n",
    "\n",
    "        # 保存变量\n",
    "        self.attn_model = attn_model\n",
    "        self.hidden_size = hidden_size\n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        self.dropout = dropout\n",
    "\n",
    "        # 定义网络层\n",
    "        self.embedding = nn.Embedding(output_size, hidden_size)\n",
    "        self.embedding_dropout = nn.Dropout(dropout)\n",
    "        self.gru = nn.GRU(hidden_size, hidden_size*2, n_layers, dropout=dropout)\n",
    "        self.concat = nn.Linear(hidden_size * 4, hidden_size)\n",
    "        self.out = nn.Linear(hidden_size, output_size)\n",
    "        \n",
    "        # 选择注意力计算方法\n",
    "        if attn_model != 'none':\n",
    "            self.attn = Attn(attn_model, hidden_size)\n",
    "\n",
    "    def forward(self, input_seq, last_hidden, encoder_outputs):\n",
    "        # 注意：我们encoder一次计算所有时刻的数据，但是decoder我们目前还是一次计算一个时刻的（但是是一个batch）\n",
    "        # 因为Teacher Forcing可以一次计算但是Random Sample必须逐个计算\n",
    "        # 得到当前输入的embedding \n",
    "        batch_size = input_seq.size(0)\n",
    "        embedded = self.embedding(input_seq)\n",
    "        embedded = self.embedding_dropout(embedded)\n",
    "        embedded = embedded.view(1, batch_size, self.hidden_size) # S=1 x B x N\n",
    "\n",
    "        # 计算gru的输出和新的隐状态，输入是当前词的embedding和之前的隐状态。\n",
    "        rnn_output, hidden = self.gru(embedded, last_hidden)\n",
    "        \n",
    "        \n",
    "        # 根据当前的RNN状态和encoder的输出计算注意力。\n",
    "        # 根据注意力计算context向量 \n",
    "        attn_weights = self.attn(rnn_output, encoder_outputs)\n",
    "        context = attn_weights.bmm(encoder_outputs.transpose(0, 1)) # B x S=1 x N\n",
    "\n",
    "        # 把gru的输出和context vector拼接起来\n",
    "        rnn_output = rnn_output.squeeze(0) # S=1 x B x N -> B x N\n",
    "        context = context.squeeze(1)       # B x S=1 x N -> B x N\n",
    "        concat_input = torch.cat((rnn_output, context), 1)\n",
    "        concat_output = torch.tanh(self.concat(concat_input))\n",
    "\n",
    "        # 预测下一个token，这里没有softmax，只有计算loss的时候才需要。\n",
    "        output = self.out(concat_output)\n",
    "\n",
    "        # 返回最终的输出，GRU的隐状态和attetion（用于可视化）\n",
    "        return output, hidden, attn_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试模型\n",
    "\n",
    "为了初步验证上面的代码是否正确，我们用少量的数据来跑一下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_batches torch.Size([7, 3])\n",
      "target_batches torch.Size([6, 3])\n"
     ]
    }
   ],
   "source": [
    "small_batch_size = 3\n",
    "input_batches, input_lengths, target_batches, target_lengths = random_batch(small_batch_size)\n",
    "\n",
    "print('input_batches', input_batches.size()) # (max_len x batch_size)\n",
    "print('target_batches', target_batches.size()) # (max_len x batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建一个较小的模型用来验证代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "small_hidden_size = 8\n",
    "small_n_layers = 2\n",
    "\n",
    "encoder_test = EncoderRNN(input_lang.n_words, small_hidden_size, small_n_layers)\n",
    "decoder_test = LuongAttnDecoderRNN('general', small_hidden_size, output_lang.n_words, small_n_layers)\n",
    "\n",
    "if USE_CUDA:\n",
    "    encoder_test.cuda()\n",
    "    decoder_test.cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoder_outputs torch.Size([7, 3, 16])\n",
      "encoder_hidden torch.Size([2, 3, 16])\n"
     ]
    }
   ],
   "source": [
    "encoder_outputs, encoder_hidden = encoder_test(input_batches, input_lengths, None)\n",
    "\n",
    "print('encoder_outputs', encoder_outputs.size()) # max_len x batch_size x hidden_size\n",
    "print('encoder_hidden', encoder_hidden.size()) # n_layers * 2 x batch_size x hidden_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后我们从SOS开始，然后用decoder来逐个解码。和encoder不同，encoder是一次处理整个序列，但是decoder必须逐个解码，因为在预测（或者random sampling时）都必须把上一个时刻的输出作为当前时刻的输入。所以前一个时刻没有处理完是没有办法处理下一个时刻的数据的。不过我们仍然是一次处理一个batch的数据的，比如之前的效率还是会提高。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 6.3927531242370605\n"
     ]
    }
   ],
   "source": [
    "max_target_length = max(target_lengths)\n",
    "\n",
    "# decoder的第一个输入是SOS\n",
    "decoder_input = Variable(torch.LongTensor([SOS_token] * small_batch_size))\n",
    "decoder_hidden = encoder_hidden # 把第一层的encoder的输出作为decoder的输入。\n",
    "all_decoder_outputs = Variable(torch.zeros(max_target_length, small_batch_size, decoder_test.output_size))\n",
    "\n",
    "if USE_CUDA:\n",
    "    all_decoder_outputs = all_decoder_outputs.cuda()\n",
    "    decoder_input = decoder_input.cuda()\n",
    "\n",
    "# 遍历Run through decoder one time step at a time\n",
    "for t in range(max_target_length):\n",
    "    decoder_output, decoder_hidden, decoder_attn = decoder_test(\n",
    "        decoder_input, decoder_hidden, encoder_outputs\n",
    "    )\n",
    "    all_decoder_outputs[t] = decoder_output # Store this step's outputs\n",
    "    decoder_input = target_batches[t] # Next input is current target\n",
    "\n",
    "# 通过mask计算交叉熵\n",
    "loss = masked_cross_entropy(\n",
    "    all_decoder_outputs.transpose(0, 1).contiguous(),\n",
    "    target_batches.transpose(0, 1).contiguous(),\n",
    "    target_lengths,\n",
    "    USE_CUDA\n",
    ")\n",
    "print('loss', loss.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 训练\n",
    "\n",
    "### 一次训练\n",
    "\n",
    "对于一个训练数据，我们首先用encoder对输入句子进行编码，得到每个时刻的输出和最后一个时刻的隐状态。最后一个隐状态会作为decoder隐状态的初始值，并且我们会用一个特殊的`<SOS>`作为decoder的第一个输入。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(input_batches, input_lengths, target_batches, target_lengths, encoder, decoder, \n",
    "          encoder_optimizer, decoder_optimizer, criterion, max_length=MAX_LENGTH):\n",
    "    \n",
    "    # 梯度清空\n",
    "    encoder_optimizer.zero_grad()\n",
    "    decoder_optimizer.zero_grad()\n",
    "    loss = 0 # Added onto for each word\n",
    "\n",
    "    # 进行encoding\n",
    "    encoder_outputs, encoder_hidden = encoder(input_batches, input_lengths, None)\n",
    "    \n",
    "    # 准备输入和输出变量\n",
    "    decoder_input = Variable(torch.LongTensor([SOS_token] * batch_size))\n",
    "    decoder_hidden = encoder_hidden\n",
    "    #decoder_hidden = decoder_hidden.contiguous()\n",
    "    \n",
    "    max_target_length = max(target_lengths)\n",
    "    all_decoder_outputs = Variable(torch.zeros(max_target_length, batch_size, decoder.output_size))\n",
    "\n",
    "\n",
    "    if USE_CUDA:\n",
    "        decoder_input = decoder_input.cuda()\n",
    "        all_decoder_outputs = all_decoder_outputs.cuda()\n",
    "\n",
    "    # 解码\n",
    "    for t in range(max_target_length):\n",
    "        decoder_output, decoder_hidden, decoder_attn = decoder(\n",
    "            decoder_input, decoder_hidden, encoder_outputs\n",
    "        )\n",
    "\n",
    "        all_decoder_outputs[t] = decoder_output\n",
    "        decoder_input = target_batches[t] # 当前输出是下一个时刻的输入。\n",
    "\n",
    "    # 计算loss和反向计算梯度\n",
    "    loss = masked_cross_entropy(\n",
    "        all_decoder_outputs.transpose(0, 1).contiguous(), # -> batch x seq\n",
    "        target_batches.transpose(0, 1).contiguous(), # -> batch x seq\n",
    "        target_lengths,\n",
    "        USE_CUDA\n",
    "    )\n",
    "    loss.backward()\n",
    "    \n",
    "    # clip梯度\n",
    "    ec = torch.nn.utils.clip_grad_norm_(encoder.parameters(), clip)\n",
    "    dc = torch.nn.utils.clip_grad_norm_(decoder.parameters(), clip)\n",
    "\n",
    "    # 更新参数\n",
    "    encoder_optimizer.step()\n",
    "    decoder_optimizer.step()\n",
    "    \n",
    "    return loss.item(), ec, dc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 进行训练\n",
    "\n",
    "所有的准备工作都就绪，我们可以开始初始化网络和开始训练。\n",
    "\n",
    "我们首先初始化模型，优化器，损失函数，以及一些用于绘图和记录进度的变量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 模型的配置\n",
    "attn_model = 'dot'\n",
    "hidden_size = 500\n",
    "n_layers = 2\n",
    "dropout = 0.1\n",
    "batch_size = 100\n",
    "\n",
    "# 训练的超参数\n",
    "clip = 50.0\n",
    "teacher_forcing_ratio = 0.5\n",
    "learning_rate = 0.0001\n",
    "decoder_learning_ratio = 5.0\n",
    "n_epochs = 5000\n",
    "epoch = 0\n",
    "plot_every = 20\n",
    "print_every = 100\n",
    "evaluate_every = 500\n",
    "\n",
    "# 初始化模型\n",
    "encoder = EncoderRNN(input_lang.n_words, hidden_size, n_layers, dropout=dropout)\n",
    "decoder = LuongAttnDecoderRNN(attn_model, hidden_size, output_lang.n_words, n_layers, dropout=dropout)\n",
    "\n",
    "# 初始化optimizers和criterion\n",
    "encoder_optimizer = optim.Adam(encoder.parameters(), lr=learning_rate)\n",
    "decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate * decoder_learning_ratio)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "# 把模型放到GPU上\n",
    "if USE_CUDA:\n",
    "    encoder.cuda()\n",
    "    decoder.cuda()\n",
    "\n",
    "# 计时\n",
    "start = time.time()\n",
    "plot_losses = []\n",
    "print_loss_total = 0 # 每过print_every次清零\n",
    "plot_loss_total = 0 # 每过plot_every次清零"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一些工具函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def as_minutes(s):\n",
    "    m = math.floor(s / 60)\n",
    "    s -= m * 60\n",
    "    return '%dm %ds' % (m, s)\n",
    "\n",
    "def time_since(since, percent):\n",
    "    now = time.time()\n",
    "    s = now - since\n",
    "    es = s / (percent)\n",
    "    rs = es - s\n",
    "    return '%s (- %s)' % (as_minutes(s), as_minutes(rs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 预测(翻译)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(input_seq, max_length=MAX_LENGTH):\n",
    "    input_seqs = [indexes_from_sentence(input_lang, input_seq)]\n",
    "    input_lengths=[len(sen) for sen in input_seqs]\n",
    "    with torch.no_grad():\n",
    "        input_batches = Variable(torch.LongTensor(input_seqs)).transpose(0, 1)\n",
    "    \n",
    "    if USE_CUDA:\n",
    "        input_batches = input_batches.cuda()\n",
    "        \n",
    "    # 预测的时候不需要更新参数\n",
    "    encoder.train(False)\n",
    "    decoder.train(False)\n",
    "    \n",
    "    # encoding\n",
    "    encoder_outputs, encoder_hidden = encoder(input_batches, input_lengths, None)\n",
    "\n",
    "    # 创建开始的SOS\n",
    "    with torch.no_grad():\n",
    "        decoder_input = Variable(torch.LongTensor([SOS_token])) # SOS\n",
    "    decoder_hidden = encoder_hidden\n",
    "    #decoder_hidden = decoder_hidden.contiguous()\n",
    "    \n",
    "    if USE_CUDA:\n",
    "        decoder_input = decoder_input.cuda()\n",
    "\n",
    "    # Store output words and attention states\n",
    "    decoded_words = []\n",
    "    decoder_attentions = torch.zeros(max_length + 1, max_length + 1)\n",
    "    \n",
    "    # Run through decoder\n",
    "    for di in range(max_length):\n",
    "        decoder_output, decoder_hidden, decoder_attention = decoder(\n",
    "            decoder_input, decoder_hidden, encoder_outputs\n",
    "        )\n",
    "        decoder_attentions[di,:decoder_attention.size(2)] += decoder_attention.squeeze(0).squeeze(0).cpu().data\n",
    "\n",
    "        # Choose top word from output\n",
    "        topv, topi = decoder_output.data.topk(1)\n",
    "        ni = topi.item()\n",
    "        if ni == EOS_token:\n",
    "            decoded_words.append('<EOS>')\n",
    "            break\n",
    "        else:\n",
    "            decoded_words.append(output_lang.index2word[ni])\n",
    "            \n",
    "        # Next input is chosen word\n",
    "        decoder_input = Variable(torch.LongTensor([ni]))\n",
    "        if USE_CUDA: decoder_input = decoder_input.cuda()\n",
    "\n",
    "    # Set back to training mode\n",
    "    encoder.train(True)\n",
    "    decoder.train(True)\n",
    "    \n",
    "    return decoded_words, decoder_attentions[:di+1, :len(encoder_outputs)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "我们可以随机选些训练集中的句子来看模型的预测和参考翻译的差异："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_randomly():\n",
    "    [input_sentence, target_sentence] = random.choice(pairs)\n",
    "    evaluate_and_show_attention(input_sentence, target_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 注意力的可视化\n",
    "\n",
    "注意力机制的一个优点是它的输出的可解释性。因为它在decode每一步都会对所以输入赋予不同的权重，因此我们可以认为它在翻译这个词的时候”对应“了源语言的词的权重，这可以认为是一种对齐。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Setting up a new session...\n"
     ]
    }
   ],
   "source": [
    "import io\n",
    "import torchvision\n",
    "from PIL import Image\n",
    "import visdom\n",
    "vis = visdom.Visdom()\n",
    "\n",
    "def show_plot_visdom():\n",
    "    buf = io.BytesIO()\n",
    "    plt.savefig(buf)\n",
    "    buf.seek(0)\n",
    "    attn_win = 'attention (%s)' % hostname\n",
    "    vis.image(torchvision.transforms.ToTensor()(Image.open(buf)), win=attn_win, opts={'title': attn_win})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了更好的可视化效果，我们下面设置colorbar和axes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_attention(input_sentence, output_words, attentions):\n",
    "    # Set up figure with colorbar\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111)\n",
    "    cax = ax.matshow(attentions.numpy(), cmap='bone')\n",
    "    fig.colorbar(cax)\n",
    "\n",
    "    # Set up axes\n",
    "    words=input_lang.seg_words(input_sentence)\n",
    "    ax.set_xticklabels([''] + words + ['<EOS>'])\n",
    "    ax.set_yticklabels([''] + output_words)\n",
    "\n",
    "    # Show label at every tick\n",
    "    ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "\n",
    "    show_plot_visdom()\n",
    "    plt.show()\n",
    "    plt.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_and_show_attention(input_sentence, target_sentence=None):\n",
    "    output_words, attentions = evaluate(input_sentence)\n",
    "    output_sentence = ' '.join(output_words)\n",
    "    print('>', input_sentence)\n",
    "    if target_sentence is not None:\n",
    "        print('=', target_sentence)\n",
    "    print('<', output_sentence)\n",
    "    \n",
    "    show_attention(input_sentence, output_words, attentions)\n",
    "    \n",
    "    # Show input, target, output text in visdom\n",
    "    win = 'evaluted (%s)' % hostname\n",
    "    text = '<p>&gt; %s</p><p>= %s</p><p>&lt; %s</p>' % (input_sentence, target_sentence, output_sentence)\n",
    "    vis.text(text, win=win, opts={'title': win})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 放到一起\n",
    "\n",
    "接下来就是开始训练了，注意需要在先运行 python -m visdom.server"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3m 46s (- 185m 17s) (100 2%) 3.1915\n",
      "7m 22s (- 176m 50s) (200 4%) 1.7054\n",
      "10m 51s (- 170m 3s) (300 6%) 0.8723\n",
      "14m 21s (- 165m 6s) (400 8%) 0.4050\n",
      "17m 49s (- 160m 23s) (500 10%) 0.2019\n",
      "> 我 知道 我 欠 你 钱 。\n",
      "= i know i owe you money .\n",
      "< i know i owe you money . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAD8CAYAAAAMs9NCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAGMBJREFUeJzt3XuUZWV95vHv0w1NQzcgCAJNMhAXDJFL0JlOQEAprjZjVMQIRIXBS3oZFeJyeUHBRFcAkahhJnJJB0ijIEpcCfFCVBhoQSENXUQyYBAZBJSZBlrBtgFRu575491FH8u6nKq965xTm+fD2qtPnfPW+9unqPqd97L3+8o2ERFtNa/fJxARMZuS5CKi1ZLkIqLVkuQiotWS5CKi1ZLk5hhJW0hSv89jrpG0fcfjnSVt08/zid5JkpuCpM0k/cE0yu/U8fiFDZ3DfElDkt4BfAg4VtLRTdTdT5L2lLRQ0laSdp7FOEuB/9Hx1GuBP5T0CUm7zVbcGAytTHINt3ZeApzUUfdWVUtg9NhC0sclLaqK/Jmk4yRtB3xW0vyZBpa0taSvArdSktv3gV8CPwfeKultM35Xk8e9RtKS2ah7jE8DvwMcA3x4NgJUP/9PArtJul7SjsBPgQ3AF4AjZyNu0yTtJOnwGt//OkkLmjynuWKzfp9AU6pf5pcBewM7Ad+R9KTtb9Ss+jXA4ZJWAfOBL1P+ML4LHAycCWxRlTkZ2BE4iPJHuwXwDUnX2/7YDGI/CRxbxRmyfV3VKvklcDLwXyTNsz1S5w12kvS7wALb/7epOieIsx+w1vZ/SPoksKT6GY862fZDDYRaDlxu+zJJ84BXA8cD2wA3ADc2EGNWVa3cvwIel3QB8Ej10ruBtcBngcXAzbbfL2kXYCWwNXCT7dOBe4ALJb3T9jO9fg/9pLl+x4OkrYHPU5LLE5RfhgOAYeAU4Drbl8yw7s0pfwhHAP9i+whJbwd+bnulpHOBrwH/DjwFLKD8Ue0F7AF8Fbjf9j/OMP6RwHuBbYHnA/cCu1NacmspH1L/0/a1M6l/gphXUFpXv+x4ejGw0fYBDcXYHLiF8vO5Hngb5ee2A/A4cKPtAxuKNY/y//A/A1cDF1Vx/huwCngn8AHb32siXtOqFvW5wGmUpHaf7Ss6Xv9b4E7K+/o25ffl1cAdtq+W9DXgFNtrqw+w9wLPqUTXhu7qaGvn/cAttq8DNrKptXN39Ys+E1sA7wEWAk9PUOaFwD8Ah1M+OV9L6QadTelenjzD2Ni+3vYy4IPA54DXA+cBFwAnAD9qOMEdAmxt+2DKh8VFtoeq2Jc2FYfS/V9XPX47cAawC/BxYEvg9AZjzQPeCvwj5Q/8tygJ4wRKMj9zgBPcrsA5wLtsPzFBsQMorTVTPjh+H3gYeJOkXW0vs70WwPY9lJ/xBZIWzv47GAxt6K4eTkdrp+rO7Q68jo7WDjCTZLA/5ZdsAWVM5ybgfkoLYNRDwJWULvIIpRu0b/XalpSEOyPVp/iRwDJgKbAbcDFwHOU9r55p3RN4FXBq9fjFlOT+BeDrwA8bjLOS8nMcsv2mahLlccrP6mjgugZjHQv8KSWh/Q7l/+c7gMuA71HGOt/QYLwmDVFaZOs7njujYyz2CMoH65PV109Rfv/OAn4F3Chppe1zOr7/PuAZyrDOHbN47gNjzrfkZrO1Y/vbtg+l/FF+1vbLgX8dU+xXlOQ22u/fhfKHdSzwyo7nZ2Jn4LermJ+z/WbKL+argH1t/12Nun+D7Q90jIMdBhwhSbbvHvOHVjfOCOXnhqTnUVoXo3+oG/j1mdC6sb5IacldC5xQ/T/9MSWpfpjSkmyEin2bmvSyfSWwXtKpHU+fbXuoOjYC6ynDCQCLqq/3pbS89wdeIenQ6vzmA+cDV9h+TiQ4aEGSk7SkGvBfTvlEvpDyCT06/tBEa+c1wFOSPlh9/T5J1wMnjim3GfCt0V9C4A+p8TO2fYfts4H/DbhKCOdSWjz3SHqRpONnWv9EJL2R0tW+htLSmU2XUbqn6wFsfwXYthqPbMrmlFbc5yVtU12K81vAY8B2kg5uKM5plPHY0xqqD9srgXWS3j9BkdXAUJVYDwZuo0yGvdT205Rx3IXVOOgFlA/rW5s6v7lgzic5Zrm1I+nNwPW2P1rFej7wV7aPpEx4LAD2o8yoPh9YXF2qcD3wRcosbJ34hwB/TRlcvhT4d9v7UT69r6VGd3icWJtJ+gBlIuB9wCeAF0m6rKlr/jrMA0QZ9P86pWs/6i3ANxuMtSvwPOC/A5dUcY8Gbge+QWkBNWFzyqTGTxuqDwDbV1G6mSdSuqurquME4COUD+FbKbOrtwJ/CXysGl7ZnNL9Px242PaaJs9tLpjzs6ujJB0GvJzS1fkw5Zf4GkoXdj/bV8+gzi2AT1EGfl0997vAL2zfXw0MH0mZ2b0a+ADlk3xhdSyojnNsf2aG72sh5ZKO3+guStrO9uMzqXecul4IfAX4EvAXnbNvkk6iDAe8x/bXGop3OHCk7Q9J2ouSbM5qugveEW+B7V+M8/wi20+O9z0zjPMC2482VV/U15okF/VJ2sX2/+tT7M1s/6ofsaPdkuQiotXaMCYXETGhViY5ScsTK7EGIV5bY80lrUxylMtJEiuxBiFeW2PNmmoxgpsneX1zSV+RdIukt0xVX1uTXETMQdXqPZcz+WU9pwJrbB9EWTJr60nrHOSJhx122MG77777tL/vscceY8cdd5zW9wwPD087TkRLrLM9vT+YDsuWLfO6deumLggMDw/fTVlgYtQK2ytGv1BZzFTAP1cX1P8GSV8CTrf9XUnvBYZtT7iazEDfu7r77ruzZk1vrl1s6E6ciLnowTrfvG7dOm6//fauys6bN+/ntpdO9Pro9aBT/D0uoixCAOVOmZ0mKTvYSS4i5oaR3vYIN1DukPkp5c6fDZMVzphcRNRiwHZXR0OGgUOqx/sDD0xWOC25iKjJuNZiOxOrbv/b2/anO56+HLhW0uhK4JMuwpEkFxH1GDaONJvkRicdbN9AWdm587UHJR1Fac39ebXk1ISS5CKiFtPzMTmqPUi6WnQjSS4iahvkS9GS5CKitiS5iGgt2z3vrk5HklxE1DbILbm+XCcn6fx+xI2I5hnYaHd19ENfWnK2392PuBExO9KSG0PSqkleWy5pjaQ1jz32WA/PKiJmaqQal5vq6IeBu63L9grbS20vne5KIhHRB13e0tWv1l4mHiKiltF7VwdVklxE1LZxZKTfpzChJLmIqGn2btBvQr9mV4f6ETcimmdDw/fnNyotuYioLWNyEdFqSXIR0Vr9WGppOpLkIqIeO7OrEdFu6a5GRGsZcglJRLRbLiGZoeHh4Z5t+tzL5nY2so62SXc1IlotSS4iWsuZXY2ItktLLiJaKxcDR0Tr5RKSiGi1XEISEa1lm5FMPEREm2VMLiJaLbOrEdFqSXIR0Vru456q3ZhRkpN0CoDtlU2eTETMTbmEJCJay8DGAb6GZF6db5a0j6QbJH1S0tmSvinpO5J2lrSFpKuq566UtEDS7ZJeIOkRSbtIurapNxIR/WO7q6Mf6iS5XYArgTcAPwP2sH0o8DngcOBPgLuq5+4F3gL8AHgFcBtwNHDH2EolLZe0RtKaGucWET00Uo3LTXV0Q9Klkm6RdOYEr28n6VpJN0u6eKr66iS5dwE/Anarvv5M9e+jwAJgb2B19dxq4EWUpHY88FXg9cDw2Eptr7C91PbSGucWEb3SZSuum5acpOOA+bYPApZI2nOcYicBV9h+GbC1pElzRZ0k95fAO6p/AZ4c8/rdwIHV4wOrr/8NOAy4jtKi+42WXETMLWZa3dUdRntq1bF8THVDwNXV4xuAQ8YJ+WNgL0nPA34beGiy86sz8fBz2w9Jugd4NbBqzOuXACsl3QT8EDgH2LY6oR8Aj9p+sEb8iBgQ07iEZN0UvbRFwMPV4/XAHuOU+RbwSuA04B7g8ckCzijJdV46Yvu0iV4D/njMtz5G6cYC7DqT2BExeBq8Tm4DsGX1eDHj9zbPAd5ue72k9wBvBlZMVGGt2dWIiNH15BqaeBhmUxd1f+CBccpsBewnaT5wQHUKE0qSi4h6Gpx4AK4BTpL0Kcok5d2SzhpT5mOUlttPge2BqyarMBcDR0RtTXVXqy7oEHAUcJ7ttcCdY8rcBuzTbZ1JchFRy+jsamP12Y+zaYa1tiS5iKgtu3VFRIs5N+hHRHvZ5RhUSXKVRYu27Vmsn2zY0LNYANsvXtzTePHc07r15CIiOmVl4IhorWwuHRHtli0JI6L10pKLiDbzAC9/niQXEbUNcEMuSS4i6inXyQ1ulkuSi4jakuQiosXMyMbMrkZES6W7GhGtN8hJri8rA0s6vx9xI2KWjN6lP9XRB31pydl+dz/iRsTsGOCGXN9acqv6ETciZoHLxEM3Rz8M3Jhctdns2A1nI2JANb38edMGLsnZXkG1h6Kkwf3JRcSzkuQiotWS5CKivWzIDfq/zvZQP+JGxOxISy4iWsvASFpyEdFaua0rItoui2ZGRIs5LbmIaLckuYhorSy1FBGt541JchHRYmnJzQFPPbW+Z7G2X7y4Z7Ggt7+AknoWKwaEM/EQES2XJBcRrZWlliKi3Qwe4N26+rIycES0SRmT6+bohqRLJd0i6cwpyl0o6VVT1ZckFxG1NbWPjaTjgPm2DwKWSNpzgnIvA3a2/eWp6kySi4japtGS20HSmo5j7FYHQ8DV1eMbgEPGxpK0OfB3wAOSXjPVuWVMLiJqsad1g/4620sneX0R8HD1eD2wxzhlTga+C5wHnCrpP9n+m4kqTEsuImprcExuA7Bl9Xgx4+eolwArbK8FrgAOm6zCJLmIqMmMjIx0dXRhmE1d1P2BB8Ypcx/wwurxUuDBySpMdzUi6mn2Bv1rgJslLQGOAU6UdJbtzpnWS4HLJJ0IbA780WQVJslFRH0NLZppe72kIeAo4LyqS3rnmDI/A17fbZ1JchFRS7njocH67MfZNMNaWyNjcpK2kHSVpG9KulLSv0l6gaRHJO0i6VpJW0n6oqSbJF3QRNyIGAxNXgzctKYmHv4EuMv2ocC9wK7AK4DbgKOBO4DlVZmXA7tI+r3xKpK0fPQamobOLSJmk83IxpGujn5oKsntDayuHq8GtgWOB75K6TsPA3sBr5W0ijIzsut4FdleYXvpFNfSRMQAeS605O4GDqweHwicRrl25TpKi+4O4HvA+dXG0mcCDzUUOyL6aHQVkrYnuUuAfSTdBOxJmQZ+CPgB8KjtBym3YRxTlXk78MOGYkdEP43OPDRx8+osaGR21fYzwB+PeXrv6t9dqzJPUrqwEdEqWRk4IlrOg7ucXJJcRNRkur1lqy+S5CKilix/HhGtlyQXES3m6awn13NJchFRT7OrkDQuSS4i6kuSi4i2MjCS7mr0k6Sexeplt6WX7ysmMb09HnouSS4iasodDxHRcklyEdFqSXIR0Vo2uE8LYnYjSS4iahvghlySXETUlYmHiGi5JLmIaK/c1hURbWZyMXBEtJpxFs2MiNZKdzUi2m6Ac1ySXETU1/oxOUkfBe6xfZWkvwAeAJYBS4AfAW8GPgSssr1K0ikAtleOU9dyYHkT5xURs2/Q93hoanPpz7Bp39VlwNbAXbYPBe4F3tJtRbZX2F5qe2lD5xYRs6kak+vm6IdGkpzt/wNsLWkIuIuysfTq6uXVwIvGfMuWTcSNiEFgRkZGujr6oamWHMDngcsorbq7gQOr5w+svv4FpYUHpbUXES3hEXd19EOTSe6LlO75t4BLgH0k3QTsCawEvgS8T9LFwI8bjBsR/VQG5bo7+qCpiYd9gL8HznHpeD/DpjG6UXcBL28iXkQMjtEcN6gaSXK27wb+oIm6ImLuaXJSQdKllHH8a22fNUm5nYCv2X7JZPU12V2NiOcim5GNI10dU5F0HDDf9kHAEkl7TlL8E3QxiZkkFxG1TeMSkh0krek4xl4TOwRcXT2+AThkvHiSDgeeBNZOdW654yEiapnmxcDrprgGdhHwcPV4PbDH2AKSFgB/DhwLXDNVwCS5iKitwTG5DWzqgi5m/N7m6cAFtp/oZu/ddFcjoqYuLx/pLhEOs6mLuj/lFtGxjgTeKWkV8GJJl0xWYVpyEVGPwc3dzHANcLOkJcAxwImSzrJ95rPh7GcvRZO0yvbbJqswSS4a1U33oSm9vBeyl+9rLmrqli3b66vbQ48CzrO9FrhzkvJDU9WZJBcRtTS9Contx9k0w1pbklxE1JOVgSOi3fp38303kuQior605CKizUySXES0lG1GRjb2+zQmlCQXEbVl4iEiWi1JLiJaLUkuIlqrLKPUn01qupEkFxG1JclFRKuluxoRrTZnk5ykYeBRyp6pO1Numj0M2BYYtv1uSSuB+ymrBswHjgC2oWxDuC3wZdsfk3Q5cLHtWyX9PfC3tv91nJjLgbFLIkfEwBrsMbmpFs3cCng98HvAG4B9gatsvwzYVtLoJtGLq+e+A7wE+CDwhWozimMlPZ+y6fQbqqWL9x4vwQHYXmF76RRLJEfEgLCntcdDz02V5B6xvQF4ENgI7ASsrl5bTdk2DODy6t9HgQXAXsCfVit3LgKWADcCLwVeSdloOiJaYpCT3HTH5J4EDgTuq/79HGWJ4ifHlPse8M+2b5T0JuAntkckXUfZRuyweqcdEYPDuKFFM2fDdPd42IKyHPG3gCdsf2OCcucC75X0bWAZ8Ej1/D8AD9l+aEZnGxEDyYx0dfTDpC250aWFO5YYfvU4ZU7pePyRjpde2VlO0hHAecAZMznRiBhcc3Z2tUm2/xfwX3sVLyJ6w1kZOCLarX+TCt1IkouI2rKeXES0WlpyEdFeZVCu32cxoSS5iKjFZI+HiGi5Qb53NUku5ixJPYvV6zGnXr63+jK7GhEtNzLAt3UlyUVELWXeIUkuIlor3dWIaLskuYhos1xCEhGtlu5qRLSW7dy7GhHtlpZcRLRaklxEtFqTSU7SpZRNsq61fdY4r28LfJ6SvzYAJ9j+xUT1TXePh4iIMQwe6e6YgqTjgPnVdqZLJO05TrE3Ap+yfRSwlrKPzITSkouIWmwY6f6Ohx0kren4eoXtFR1fD1E2sQe4ATgE+P6vx/OFHV/uSNkKdUJJchFR2zS6q+um2Dh+EfBw9Xg9sMdEBSW9FNhuoo3qRyXJRURNbvLe1Q3AltXjxUwwpCZpe+BvgNdNVeHAjclJWi5pzZgmbUQMMNtdHV0YpnRRoWxc/8DYApIWULq0H7T94FQVDlySs73C9tIpmrQRMUAaTHLXACdJ+hRwPHC3pLEzrG+lbG96hqRVkk6YrMJ0VyOilib3XbW9XtIQcBRwnu21wJ1jylwEXNRtnT1vyUnae5zMHBFzlrE3dnV0VZv9uO2rqwRXW89bcra/C5zZ67gRMXtyx0NEtFqSXES0WFYGjogWyx4PEdF6aclFRIsZZ0vCiGiz7PEQEa2WMbmIaK0m73iYDUlyEV2Q1NN4vUwa9d9bLiGJiJYbycRDRLRZxuQior3KoFy/z2JCSXIRUYvJJSQR0XKZeIiIVsuYXES0mDO7GhHtlYuBI6L1BjnJTWuPB0k7STp8psEkva7aTiwiWsPgke6OPug6yUnaGfgEcKyk/6i2Alsl6cWSdpZ0naRbJZ1Xld9F0tcl3SLp3Kqae4ALJW3R/FuJiH5xl//1Q1dJTtIS4DzgVOAnwNm2h6rjO8BHgX8CDgIOkXQQ8GfApbYPAl4saWfbd1MS5QVJdBHt0eC+q42bMslJ2hU4B3iX7ScmKHYAcJPLu7gF+H3gYeBNkna1vWx0ezHb9wAfpyS6hU28iYjoH9uMjGzs6uiHblpyQ8Adttd3PHdGR3d1PrA18GT12lPANsCngX8BbpT0oTF13gc8A+w9Npik5ZLWSFozvbcSEf0yp1tytq8E1ks6tePpzu7qRmA9sLh6bVH19b7ApcD+wCskHQpQJcXzgSts3zFOvBW2l9peWueNRUTvzOkkB2B7JbBO0vsnKLIaGFJZmOpg4DbKBtIvtf00cC+wUNLmwAXAZ23fWvfkI2IwDHKS6/o6OdtXSTqOMj5nSW+rXroI+AhwBfBG4Gbbt0r6GbBC0q+A+4HrgDOAi6vJiohoiwG+Tk6DfBGfpME9uYhZ1OOVgYfrDA/NmzffCxdu1VXZp5/eUCvWTOSOh4iobZAbS0lyEVFbklxEtFg2somIlst6chHRWllqKSJazgPdkpvWUksREeOxR7o6uiHp0mr1ojPrlBmVJBcRtTV1x0N1w8H8avWiJZL2nEmZToPeXV0HPDiD79uh+t5eSKy5FavX8WYUq9wh2ZtYwG4zCdbh67Z36LLswjGLb6ywvaLj6yHg6urxDcAhwPfH1NFNmWcNdJKzveNMvk/Sml5dVZ1YcytWr+O1NVYn28sarG4RZZk2KAt97DHDMs9KdzUiBskGYMvq8WLGz1HdlHlWklxEDJJhSvcTyjJtD8ywzLMGurtaw4qpiyTWczRWr+O1NdZsuQa4udpy4RjgREln2T5zkjIHTlbhQK9CEhHPPZK2A46ibKmwdqZlni2bJBcRbZYxuYhotSS5iGi1JLmIaLUkuYhotSS5iGi1/w9u6/BcIgA9PAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "21m 16s (- 156m 2s) (600 12%) 0.1171\n",
      "24m 41s (- 151m 41s) (700 14%) 0.0928\n",
      "28m 8s (- 147m 43s) (800 16%) 0.0649\n",
      "31m 31s (- 143m 37s) (900 18%) 0.0541\n",
      "34m 58s (- 139m 55s) (1000 20%) 0.0421\n",
      "> 你 可以 进来 。\n",
      "= you can come in .\n",
      "< you can come in . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARUAAAD8CAYAAABZ0jAcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFSxJREFUeJzt3X+UXkV9x/H3hxAMBkU0CAarOR6oCCiWBpUYZKVEQ22PFo+gAj34o6kVAdtaRYiKioCUQ+1RkG6JRn4IpSr4i+qxxVQ0CCZWLEEUqmDFIkTBGBTQ7Kd/3LtkWXb3eXYzzz737n5eOffkPntnZ+bZH9+dmTt3RraJiChlu35XICJmlgSViCgqQSUiikpQiYiiElQioqgElYgoKkElIoravt8V6CVJewH/SxU8H2/7rmkocwA4GtgyXhLgS7avrNO/A/ii7Zvq1ycC/2n7xoJ1+gfgLNs/k7QEOBz4EvBntt82RvqTgPttX1iqDk0maTdgX9vXTPHzXwl83vZDZWvWTprJk98kfRl4K7APcKjt46ehzPnA44DfjXH5BuD5wAPAGcAHgScAnwRebPvnkq4GTrN9Q8E6vQx4LPBS4ADg8cDdwFOA/wOOAlYCewMGFgJDwF1UQfC7tk8oVZ8mkbQ78PfAvcAy4Gf1pbdSvf+LgZ2Aa22/XdJTgNVU3+Ov2T5Z0r7AXwPH235wmt9C89iekQfwbOAT9fnVwHeANSOOp/Ww7KeOKuvd9cdvG5Hms8Ci+vxo4Iz6fB3w5IJ12R6YSxUctgdeArwDWEzVehlO9xjgL4CdgTcBxwFPB94C7NDv72ePvk8LgYuoAvtpwDGjrv8T8Ob6a7cWWAKcBRxZX/8SsHt9vjdwIfCYfr+vfh8zckxF0lzgY8CPJC2l+qv8POC1VE3/ebZ/3MMqbA/cbnuA6hf0GWOkebiJaPtS4F31yyfYvrtgXY4Fvg/8FdUvx2OB+4YvStpe0nau/sJ+D/ibEZ97MvB1z8BmvaQ9qFqLb7F93zjJnk/VGjFVUDkQuBM4RtIetpe77lLbvoWq5XmepHm9fwfNNVPHVI4FNtbnb6L6y/wU4HTgJKpflkaQtAN1IJH0bOCnJfO3/XFJv6X6Xl8NLAJ+TdUF2w04CHivpPcCD1K1avakCnq3AufUQfpM218qWbc+GwC+bXvTiI+dKumN9fkfUXVx7q9f/5qq23g6Vdf2q5JW2z5jxOffRvU13Af4dg/r3mgzsqVC1ef9IIDtY4B9gSdTDZ6+BPjvaajDcklrqJrXGifNXOAy4FX169cBn+pVhWwvo+qOHQscD1xu+xDb19g+2PZhwA+Ba6laLettH1an6VlAUWU/SeN9nYqrW4ebJI0cK/qA7YH62AJsohpPAZhfv94PWAXsD7xU0iH1e5gDfAi4xPasDSgwQ4OK7SHqgVJJT6AKMMN/cTYD/9jjKgxR/XANAH9M1UIYy8XAv9s+T9JLqbpmx0h6Ui8qJelpwH62v0v1vdeIa3MkfRj4EVXguRTYQdIlkhb0oj4jnAg8s/5/2theDWyU9PZxklwPDNTB7oVUA+0rgYNs/wb4ATCvbsmdB1xs+7re17zZZmRQGeVjVN2dTQC2vwDsLOmwXhVo+8eub9Xa3ghcWd9leMyIZM8ALgAulvQuqi7aC4AzgS/Wd5G2WR1Uj6W6O/E+to7dbKZqriPp94FvAvfZfh91sLF9InALcOvwX+QemQssAH7ZwzLGZPsyqq/Dq6m6P2vq4yiqwduXA9dR3f25Dng/cKakr9X1/grVz9cFttdNd/2baMbeUpb0Iqp+8SW2b63nrJxq+zhJu1L9Av12mupyINVfsittn1l/bAHVL/Za4F+Bc+vBUiT9JfAN13NXtrHsE6hujd4MvAF4DlWTfh5Va24B8B7gJ7b/o/6cvwV+ZXuwfv0M4Efu4Q+LpCcXHqCOPpmxQaUt6jsvQ30ody7V4O12wAP1GELENktQiYiiZsOYSkRMo1kRVCSt6HcdRmpafaB5dUp92mtWBBWgaT8QTasPNK9Oqc80kbSbpGsnuD5X0hckrZX0+k75zZagEhFjkLQL8AmqyX3jOQFYZ3sJ8CeSHjdhnk0eqF2wYIEXLVq0zfncc8897Lrrrtucz/r167c5jwjb2zRzePny5d64cWPnhMD69es3UD2SMWxweKoAgKTHU81L+mw9WfNRJH0OONn2zZLeRjXT+qvjldnoZ38WLVrEunXNmU80jbPII8a1ceNGvvWtb3WVdrvttnvA9uLxrg8/+9ThZ3s+1YOUUE0i3W2ixI0OKhExtqHp7WFsBnakmvG8U/16XBlTiWgZ0/06SIWsB5bW5/sDt0+UOC2ViNYxpjctFUmHAvvY/siID38CuFrSwVTLOlw/UR4JKhFtY9gyVDaoDA/Sulqn95pR1+6QtIyqtfLuTo90JKhEtIyZ9jEVbP8UuKKbtAkqES3U5KkgCSoRLZSgEhHF2J727s9kJKhEtFBaKhFRjIEtCSoRUVJaKhFRVMZUIqKcslPwi0tQiWiZ4Wd/mipBJaKFtgxN+wYMXSsSVOp9eG+xfZmk91A9xbgcWAj8hGo7z1OANbbXSDoOHt4hLiImpXcPFJZQaumDi4DX1OfLqTa2vsn2IVRbQ3Zc13KYpBWS1klad8899xSqXsTMYcNQl0c/FAkqtv8HeJykAeAmHvl49PXAs0Z9yo4T5DVoe7HtxSWWgIyYiaZ5PZVJKblI0+VU+xZfBGyg2heY+v8NwENULRioWjMRMUWzJah8impg+uvAhcC+9SbWewGrgc8BfyfpAuDnBcuNmFWGlz7o5uiHUgO1+wIfB86oN/F+kK1jLMNuAl5UoryIWc2e+Xd/bG8Anlcir4joLPNUIqIYQ6NvKSeoRLRQv24XdyNBJaKF0v2JiKISVCKiGM+Guz8RMb3SUomIYvqx789kJKhEtFBuKUdEUbmlHBHF2GYoA7URUVLGVKZo/fr1SOp3NR7WxBH3Jn19Yvo08WdxWKODSkSMLUElIorJXsoRUVxuKUdEMQa2NPiecoJKRAs1eUyl5Bq1ETFNSq1RK2mVpLWSVo5zfRdJV0u6tl5fuqMElYi26XIl/U6tGUlHAHNsLwEWStprjGTHApfYPphqG57FnaqXoBLRMsN7KXcZVBYMb85XHytGZDUAXFGfXwMsHaO4nwPPlPQE4PeAH3eqX8ZUIlpoEreUN9oer3UxH7izPt8E7DlGmq8DLwNOBG4B7u1UYFoqES1UaExlM1t3C92JsePBGcCbbL+PKqi8rlOmCSoRLVNwM7H1bO3y7A/cPkaaxwLPljQHeH5d/IQSVCLaptBALXAVcKykc4EjgQ2STh+V5kxgEPgl8ETgsk6ZZkwlooVKTNO3vUnSALAMONv2XcCNo9LcAOw7mXwTVCJaZvjuT5G87HvZegeoiASViBbKavoRUZDzQGFElGNXR1MlqES00IxfT0XSPGA18FTgPuC1wOXAPOAO26+TdBowl+q++M7A8nq0eXReK4AVoz8eEVvNhqeUVwA32l4KfBp4FnAecDiwSNJudbo9bR8CfBI4dKyMbA/aXjzB1OKIWa3g5LeeKBVU9gZuqM9XA3cDbwQupZowMzwV+KL6/7uBHQqVHTG71Ft0dHP0Q6mgcgtwYH1+CrAW+BTwGuD+EenuJyK23fBobaejD0oFlUHgAElrgAOAo4B3Uj1ODbBHoXIiAvCQuzr6ochAre0HqJ4dGGm/Ua+/MSL96hLlRsxWDR6nzS3liLapejbNjSoJKhEtlKASEQWZoS159iciCkn3JyKKS1CJiLISVCKipAbHlASViNZxBmojoqCSy0n2QoJKRAslqEREUQkqEVGODX16WLAbCSqTIKnfVXiUpv3FauLXaCZq2vd9pASViJYxMJSWSkQUk2n6EVFavxZg6kaCSkTrdLX5et8kqES0UIJKRBSTpQ8iojhvSVCJiILSUomIcpyB2ogoLEElIorJ0gcRUZbBDV6kqdS2pxExbaoxlW6OTiStkrRW0soO6c6X9Kfd1C5BJaKFSuzPLukIYI7tJcBCSXuNk+5gYHfbn++mbgkqES00iZbKAknrRhwrRmQzAFxRn18DLB1djqS5wD8Dt0t6eTd1y5hKRMvYk3qgcKPtxeNcmw/cWZ9vAvYcI82fAzcDZwMnSHqa7Q9PVGBaKhEtVGhMZTOwY32+E2PHgz8ABm3fBVwCvLhTplNqqUiaB6wGngrcBxwNXAAsBH4CvA64DrgbeAjYHbisPlYDOwOft33mGHmvAFaM/nhEDDNDQ0Xu/qyn6vJ8E9gf+P4YaW4DnlGfLwbu6JTpVFsqK4AbbS8FPg2cBNxk+xDgB8DrgccCrwKeA7y2rvQ7gX+pB4ZeIelJozO2PWh78QRNtojZzcVaKlcBx0o6FzgS2CDp9FFpVgEvlvQ14M3AOZ0yneqYyt5UwQSqlsf5I15fDxwO/Mz2Zkl3AFsAAc8EDpJ0HFV/biHw8ynWIWL2KrBIk+1NkgaAZcDZdRfnxlFpfkXVOOjaVFsqtwAH1uen1K9fUL9+AbBhnM/7PnCy7QHgLOAXUyw/YtaqZtRu+y1lANv32r6iDihFTDWoDAIHSFoDHEDVWtm3biLtVb8ey1nA2yR9A1gO/GyK5UfMaqUmv/XClLo/th+g6oON9JpRrwfqtAP16+Pq/182lTIjopa9lCOitDxQGBHF5CnliChreKS2oRJUIlonK79FRGFu7jhtgkpE65hS0/R7IkElomUyUBsRxSWoRERBzgbtEVFQtj2NiOISVCKiFAND6f5ERDGTW6N22iWoRLROZtRGRGEJKhFRVIJKRBTjhu+lnKAS0UINbqgkqES0TwZqI6KwBJWIKCfT9COiJJPJbxFRlHEWaYqIYtL9iYjSGhxTElQi2ihjKhFRTNaojYiyGj6msl2vC5D0oV6XETG7mKGhoa6Ofuh5S8X2W3tdRsRsM6vHVCStsT1Qn58GzAWWAjsDy23f1es6RMwoDd9LuefdnzHsafsQ4JPAoaMvSlohaZ2kddNftYjmG44p3Rz90I+gclH9/93ADqMv2h60vdj24umtVkR72O7q6ETSKklrJa3skG43Sf/VTd36EVTu70OZETOHzdCWoa6OiUg6AphjewmwUNJeEyQ/B9ixm+r1I6hExDaaREtlwfBwQn2sGJHNAHBFfX4N1Vjno0g6lKox0NX453Tc/RkYcX7aiPPVvS47Yiaa5OS3jRMMJcwH7qzPNwF7jk4gaQfg3cArgKu6KTCT3yJaqNDkt81s7dLsxNg9l5OB82zfJ6mrTNP9iWidLm/9dA4869na5dkfuH2MNIcBx0taAzxX0oWdMk1LJaJtDC4zWfYq4FpJC4HDgVdLOt32w3eCbL9o+Lyec/bGTpkmqES0UIkp+LY3SRoAlgFn1xNRb5wg/UA3+SaoRLRMyaeUbd/L1jtARSSoRLRNw59STlCJaB3P7gcKI6IH0lKJiJJMgkpEFGKboaEt/a7GuBJUIlooA7URUVSCSvRMt89jTJcm/rA37WtUQhO/zsMSVCJaplrWINueRkRBCSoRUVS6PxFRVIJKRBSUMZWIKMh5oDAiSktQiYiCjPu0T3I3ElQiWsgkqEREQen+REQxGaiNiMK62ye5XxJUIloo66lERFFpqUREOd3tPtg3CSoRLWOyRm1EFJZnfyKioNz9mRRJK4AV/a5HRJOV2Eu5VxoXVGwPAoMAkpobjiP6pBqnTVCJiGKa3f3ZbroLlLSPpNOnu9yIGWX4tnKnow+mvaVi+2Zg5XSXGzGT5JZyRBTV5O5PgkpEy2Qv5YgoLi2ViCgqQSUiiioVVCStAp4FXG37UXdlJe0MXE4VKzYDR9l+aKI8p/2WckRsK4OHujsmIOkIYI7tJcBCSXuNkexo4Fzby4C7gOWdapeWSkTL2DDU/YzaBZLWjXg9WM9aBxgArqjPrwGWArc+siyfP+LlrsDdnQpMUIlooUl0fzbaXjzOtfnAnfX5JmDP8TKRdBCwi+1vdiowQSWidYpte7oZ2LE+34lxhkMkPRH4MPDKbjLNmEpEC9nu6uhgPVWXB2B/4PbRCSTtQNVFeqftO7qpW4JKRAsVCipXAcdKOhc4EtgwxnN5bwD+EDhV0hpJR3XKNN2fiJYpte+P7U2SBoBlwNm27wJuHJXmo8BHJ5NvgkpE6xi7zDR92/ey9Q5QEQkqUZSkflfhUZo0+3Tx4vFuxExOk97TaAkqES2UoBIRBTV75bcElYiWyRq1EVFcWioRUZBxtuiIiJKyRm1EFJUxlYgoptSM2l5JUIlondxSjojCspdyRBSVMZWIKKePW5p2I0ElomVMbilHRGFNHqid1MpvknaTdOhUC5P0ynp5uojYBvZQV0c/dB1UJO0OnAO8QtL36qXl1kh6rqTdJX1F0nWSzq7TP0XSlyWtlXRWnc0twPmSHlP+rUTMFmZoaKirox+6CiqSFgJnAycAvwA+YHugPr4DvBe4ElgCLJW0BDgJWFVvVPRcSbvb3kAVmM5LYImYmuHJbwXWqO2JjkFF0h7AGcBbbN83TrLnA19z9S7WAgdS7SdyjKQ9bC+v17/E9i3AB6kCy7wxylshad2oDZAiYoRWBxWqXcy+bXvTiI+dOqL7Mwd4HHB/fe3XwOOBjwD/BnxV0imj8rwNeBDYZ3RhtgdtL55gA6SIWa7Mtqe90jGo2L4U2CTphBEfHtn92UK1u9lO9bX59ev9gFVU+4m8VNIhAHUQ+hBwie1vl3srEbOHu/zXD12NqdheDWyU9PZxklwPDKha9fiFwA3ASuAg278BfgDMkzQXOA+42PZ121r5iNmqyd2fruep2L6s3iX+DMCS3lhf+ihwGnAJ1Q7x19q+TtKvgEFJvwN+CHwFOBW4oB7cjYgpsM3QUJktOnphUpPfbH8G+Mw4lw8blfYmqrtBI71/MuVFxNiaPPktM2ojWihBJSKKSlCJiLISVCKiFNsMFdpLuRcSVCJaKN2fiCgqQSUiCsrC1xFRWNaojYhisu9PRBTmRrdUJrWcZEQ0Q6nlJCWtqldnXLktaUZKUIlooRJPKdcPCM+pV2dcKGmvqaQZrendn43AHQXyWVDn1RRNqw80r07F6lOtyLHNStXn6QXy+LLtBV2mnTdqFcVB24P1+QBwRX1+DbAUuHXU53eT5hEaHVRs71oiH0nrmrSSXNPqA82rU+ozPtvLC2U1n2rZV6gWVttzimkeId2fiNlrM7Bjfb4TY8eDbtI8QoJKxOy1nqo7A9Wyr7dPMc0jNLr7U9Bg5yTTqmn1gebVKfXpvauAa+steA4HXi3pdNsrJ0jzgk6ZqsmTaCKityTtAiyj2mLnrqmmeUT6BJWIKCljKhFRVIJKRBSVoBIRRSWoRERRCSoRUdT/AzX/4UUPn1JyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "38m 29s (- 136m 26s) (1100 22%) 0.0373\n",
      "41m 55s (- 132m 44s) (1200 24%) 0.0340\n",
      "45m 23s (- 129m 10s) (1300 26%) 0.0333\n",
      "48m 55s (- 125m 47s) (1400 28%) 0.0336\n",
      "52m 23s (- 122m 14s) (1500 30%) 0.0403\n",
      "> 现在 我 需要 你 离开 。\n",
      "= now i need you to leave .\n",
      "< now i need you to leave . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAR4AAAD8CAYAAAChMsvrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAGLtJREFUeJzt3XuYZVV95vHvS3Nrmqs2t+YxdCYwPIIIGdsBW4QCQRuVDEIkJIACwR6eEZSZUUFgDBpAJIyaGC62NhBEIWgMo4giBBpQbnYTiKCIRIFIHgKdIC1XoeudP9Yuurqsyzl1du1z6tT74dlP7zp7nbX2oap+tdbae6+fbBMR0aR1un0CETHzJPBEROMSeCKicQk8EdG4BJ6IaFwCT0Q0ru8Dj6T9JG04hfVvIEk11TVL0nqS1h1lW786vu44719X0qwOz2FHSRtK2kjSNp3U1Wa764927iqm7PsX3aHpdB+PpPWBQdsvVz+kHwX+EHiqKvJ621sNK78B8CNgZ9sv13ges4C3ADsDWwP3AM/a/l6H9S4EjgQGge2q7S5AwE+q7X8BLwG/C7wArAbmAL8A1gPOsn1bB+dwHXAS5bPtZ/sDk62rqm8dANuD1dcnAw/Z/rvq6/Uon/evgN2A+cBK4FfAAuBu4CXbb+3kPKaCpK2BXWzfOMn3Hwp8y/Zv6j2z3jfdAs8+wFnA7wH3AjcAy20vq45fY/tdku4AVgGbANsD/zSsmjnAGbb/YRLtbwJcCWxJ+cX4C2APYAVwNHC97S9N8rO9Efi/lF9CgM2r7eGhIsA5tr9TlT8B+CUl+Oxg+68n0+6Ic9gV+LDt90m6FphH+ZxD3mv70Tbr3I3yuV4GtgKerA7tBDxE+Vyn2b6jKv854GrbyyTdY3v3jj7UFKl6g39B+aN3APBv1aGTgMeBLwMbA7fa/qikbYFLKT+Tt9g+RdIuwP8EPmD7xYY/QleN2W3vRbZvlnQwcJ7toyV9GEDS3wInDyu3Z/X614D3UX7o32D7ax2ewrPAwcCbgQHb10taQOmBvBf4L5LWGfrr3qaNgG8C/0oZAv8SGKD8EP+6+gybSvossBDYDHiREqhmSzoCuNf28ZP5YFXP42Lg25L2Ap6oPutcyi/XTe0GHQDb90paCrg6308CGwDfBs4DThgKOtOFpHnAOcCJlEBzlu3Lhx3/AvD3wIXAD6qe7B8AS21fJem7kraxfb+k84DzJc2o4DOtAo+kPYHPAvMlXQz8uDo0B3hsRNmTgUdtP1i9781Ap4FnP+DDlF/6V1dBZz5wKCVArEsZMlzbbsVVUL0N+BalV/Dlqr43AS/Y/u/V51oA/CWwKyU4vAi8BlgOdDIcOYoyxAE4nhLItwXOBD4EnNJB3bMo/592Bc4G/hl4P3AgMKeat1pnrCHH0LxWncPlyZK0HaXXfYLtVWNM7+0BfN62q+/pGyk/n0dK+oHtRUMFbT8g6dOU4HOC7Rca+BhdN60Cj+07JB1E6fEcK2nol+E52y8N/YBK2o/yy7JY0i2UuY9tJe1MmV/4H5Ns/wbgBkkDlN7Ip4H3UH6xrgL+0nbbQac651cDxwLfrV7amRIIVlbH/zfwPeAyylDzR8BpwMcpvSSAJZNpu3Ip8HNKT+5ISW+j9HRWA28Dru+g7jsoP2u/pPRAbwZ+UB27nNJbPFHSr4H/DOwtaRWwg6RlwGzKH5wrOziHugwAd9teNey10yQdV+2/lTKcerb6+jlgU0oAfxm4SdKlts8e9v6HKH9AdqbMafW9aRV4RvEN4Hng9yT9AXCqpHdQuvOP2r4GuKbq8Rxu+6ROGqu62PsDiygTn9sDFwGHUHpCd3ZQ/fPV+4f+6r8P+CnllxbK9+pDwO9QhldvYE3PYRfggXKKusX2J9tt3PagpJcplWxOCapHVIefofSyjmz/YwElIK5Dmc/5fUogGZpcXBd4t+2Lq7Zvrb5eWc3xDEyyTar6RPn/c79rmNC0/RVJR0s60fbnq5dHDrVWUeZ3oPTGHwVeByylBPjvVj2fm6sLFZ8BLrc9I4IOTLPL6dVVqtcDe0i6wfaDlO767cDC6hv3j8BBlF+Wum1DGdbcAXzV9jGUv1AHAa+z/cXJVmz7OeBPgM8BfwPsS5mwvpIyn/Ap28fZfhtwNWU49pGq/PdsH2B7/8kEnVFcTBlararO7RpgM0n7T7K+5yg9gZ8B77S9D3Ab8A7gPsqcz9BVvVm2V45V0SR8kDKR/cG6KrR9KbBS0kfHKHInMFAFvTdTrkyeDrzJ9vPAg8CG1bza+cCXbd9e1/lNB9Otx3M8ZZ7lVOC66krQgZS/GAdJeitlEvlcSetIWs/2S8MrqP7CaDLzBVVgu1vSvpThwObA/6EMRx6Q9FpgV9tXTfLzvQwcTulVvUy5ojUAfAK4aVi5L1GGRRdQhpE/knQ48B3bT0+ybVjTKznZ9s8k7Tjs2LGsfYWrXfsAbwdeqHqnGwN/XB2zpNdQegTHDXvPupJm2V7dQbvrUSbIO/n/8ltsXyHpEMqclYcNtS4EzqAMIY+gXNW6vRpGLql6lT+nDF1PAy6yfU+d5zYt2J6WG3AC8LesuWT+cLXtVx1fTvmh+DZwzbDtu8BBHbS7F+W+nYOBvwPeV73+Wcq9NId2UPcXKJOw76QM53an9Hr+jBJktqbMj1xXlR36rK+mBOV/AOZ10P5+wNnV/k7AI8D7a/he3UAV1Ea8fjLlVofNqu/Nu0cc/ykwu4b2t+r2z2u2tbdpdR/PcJI2cBcuP1Z30a7vtScXh45tYfupUd7WaZuv/NWvoQfQTrvregqvJEmSp+sPYHRk2gaeiJi+ptXkckT0h74MPJIWp63p01bT7fVrW9NJXwYeoMlvdtqafu31a1tTRtLW1T1WYx1fT9I1km6TdOxE9fVr4ImImkjagnJv2Zxxip1IeWB7IfCu6oHqsevs5cnluXPnev78+W2/78knn2TLLbds6z0rVqxou52IqWK7ozWeFi1a5JUrW7sPc8WKFfdTVjkYssT2K4/fSNqUcn/X//MYd5JL+iZwiu0fVw9vr7B902hlocdvIJw/fz7Lly9vpK0xHvaLmJZWrlzJD3/4w5bKrrPOOi/YXjDW8aFbRyb4HRn+oPYqyj1nY+rpwBMRkzfY7GjmGcozeE9T7kof95GlzPFE9CHT+lMJNVlBuasfykqSD49XOD2eiL5kzNT0eKplZ3b22qte/g1wraShJYHHXakhgSeiHxlWD9YbeIYmll3WmL5xxLFHJB1A6fV8fKLHehJ4IvqQaXyOB9v/SlkQb0IJPBF9qpdvlUngiehTCTwR0SjbjQ+12pHAE9Gn0uOJiEYZWJ3AExFN6+UeT213Lks6Q9JZkm6WdI+k7SVdUX39FUnrS/qhpK0k/ZukbVXS5I6sZ7Gk5ZKWP/nkk6M1FREtGKzmeSbauqHuRyZ2cEld8lVKXqj7qq8fpGQp+AUl08BdlCRxv5VHyPYS2wtsL2j3CfOIqLSx8Ho31B14Lqv+fYKSl2notuk7gddSAs1hlMwP76E83xERNevCs1ptqTvwPDts/yPAntX+nsD9lGR7+1JyCr2dGZKuNaIbVg8OtrR1w1ROLr8M7FLlLv8XSo6rzSjpXH8BPGH7kSlsP2IGm7qHROtQW+Cxfcaw/Uur3S+MKPYk5clVgO3qajsi1mZDzc+I1iqX0yP6VC9fTk/giehTCTwR0ahuLIvRjgSeiH5kd+2KVSsSeCL6VIZaEdEow8y4nB4RvSWX0ydpxYoVjSXaa7JbmuSB0YQMtSKicQk8EdEo56pWRHRDejwR0ajcQBgRXZHL6RHRuFxOj4hG2WYwk8sR0bTM8URE43JVKyIal8ATEY3q9dzpdWeZaImkz3Wj3YiZxC3+1w1d6fHYPqkb7UbMFAZW9/D19G71eJZ1o92ImWQmJfTr2PDc6d0+l4jprK7c6ZKWSrpN0uljHN9C0rWSbpV0USvn1nOBZ3ju9G6fS8S0VVPudEmHALNsLwTmSdpxlGJHAZfbfguwiaQJf3d7LvBEROfazJ0+d2iUUW2Lh1U1AFxV7d8I7DVKc/8O7CRpc+A1lGzB48rl9Ig+1cbl9JXjjDDmAI9V+6uAHUYp833gncAHgQeApyZqsCs9HtsD3Wg3YiapaY7nGWB2tb8xo8eMs4HjbX+SEniOmajSDLUi+tDQejw1BJ4VrBle7QY8PEqZjYBdJc0C9qiaH1cCT0Q/qmlyGbgaOErSZ4DDgPslnTmizKeAJcDTwKuAKyaqNHM8EX2qjkcmbK+SNAAcAJxr+3Hg3hFl7gJ2aafeBJ6IPjR0VauWuuynWHNlqxYJPBF9KlkmIqJh3XsAtBUJPBF9yC5br0rgqTSZVrjpB/OSMnlm6uX1eBJ4IvpUViCMiEYloV9ENC/pbSKiK9LjiYimuYeXPk3giehTPdzhSeCJ6EflPp7ejTwJPBF9KoEnIhpmBlfnqlZENChDrYjoil4OPF1bgVDSGdUCQxExFYaeFJ1o64L0eCL6VA93eCYXeCSdAaxHWQR6M+AdwF8BWwE/sv0BSRsBl414bQvga8AsQMCyTj9ARIzCvT253MlQawfb+wBfBY4G7rO9N7CtpNcDi8d47Rrb+wIvjVZpUhhHdK7NhH6N62SodVn17xPAhcCD1ZzN5sB2wE7AwhGv/S5r1m4dNbDYXkJZsR5JPdxZjOhtvTy53EngeXbY/seAp21fIuldlBSmPwXuGvHaI8DOlFSouwPXddB+RIyjlwNPXVe1XgIOlHQLcDzwL8AXx3jtUEnLgE1rajsiRrJhsMWtCybV47F9xrD9S6vd80cpetgor+07mTYjoj293OPJ5fSIPmRgMMtiRESj8shERHRDFgKLiIZ17x6dViTwRPSpBJ6IaFSWxYiIrvDqBJ6IaFh6PLGWpnOZN/kDmDztPaKLD4C2IoEnok8l8EREo4aWxehVCTwR/cjgPl0ILCJ6VmuLgLXSK5K0VNJtkk6foNwFkg5q5ewSeCL6VB1rvUs6BJhleyEwT9KOY5R7C7CN7W+1cm4JPBF9qo0ez9yh5YarbfGwagZYs2rojZR11tciaT3KWlsPS/pvrZxb5ngi+pDd1kOiK20vGOPYHOCxan8VsMMoZd4L/Bg4FzhR0u/Y/vx4DabHE9GnaprjeQaYXe1vzOgx4/eBJbYfBy6nhcX+Engi+pIZHBxsaZvACtYMr3YDHh6lzEPAf6r2F1DWVh9XhloR/ai+h0SvBm6VNA84EDhc0pm2h1/hWgpcLOlwSr69P5yo0gSeiH5Vw0JgtldVKaoOAM6thlP3jijza+A97dSbwBPRh8qdyzXVZT/Fmitbtagt8Ej6BPCA7Ssk/RllLLgImAf8EjgGOBVYZnuZpKNhrSwVEVGjXn5kos7J5cuAP672FwGbUFIY7wM8CBzbSiVJYRxRgyp3eitbN9QWeGz/M7BJNR68j5Ix9M7q8J3Aa0e8ZTajsL3E9oJx7iuIiBb0cu70ui+nXwlcTOn93A/sWb2+Z/X1byg9ISi9ooiYAkNPp8+UwPN1ymf+PvAlYJcqhfGOwKXAN4GPSLoI+Pea246IIUOzy50+rDVF6pxc3gW4BDjbJYy+yJo5nyH3AXvX1WZEjGWGrEBo+37gv9ZVX0R0xr27HE/u44noS6aVxyG6JoEnog9l6dOI6IoEnohomNtZj6dxCTwR/SgpjCOiKxJ4IqJJBgYz1IpuajKtcNIl94j21lxuXAJPRF+aIXcuR0RvSeCJiMYl8EREo9zjudMTeCL6VA93eBJ4IvpTJpcjogsSeCKiWXlkIiKaZnIDYUQ0zjgLgUVEo2byUEvS7gC275nKdiLit/Vw3JnyHs/u1b8JPBENm5FzPJI+Bby72j8KeAclt9YrudRt/2aU9y0GFk/VeUXMBL2+5nLdCf1eYftjwDnAObbfCryfFnKpJ4VxRA08szKJjmeiXOoRURszODjY0tYNUx14ngc2qvZHy6UeEVPEg25p64apDjzXA4dI+gElffHIXOoRMRVmSu700dj+D2D/YS/dPJXtRUQxFHd6VZNzPBHRoLomlyUtlXSbpNMnKLe1pH9s5dwSeCL6kc3g6sGWtvFIOgSYZXshME/SjuMUPw+Y3crpJfBE9Kk2ejxzJS0ftg2/j24AuKravxHYa7S2JO0HPAs83sq55VmtiD7U5g2EK8e5b24O8Fi1vwrYYWQBSesDHwcOBq5upcEEnog+VdPNgc+wZvi0MaOPkk4Bzrf9q1ZznWWoFdGXWryUPnFwWsGa4dVuwMOjlNkf+ICkZcDukr40UaXp8UT0I4PruSn5auBWSfOAA4HDJZ1p+5UrXLb3HtqXtMz2cRNVmsAT0afqeBzC9ipJA8ABwLm2HwfuHaf8QCv1JvBE9KE6n063/RRrrmzVIoEnoh/N5BUII6JbuvcAaCsSeCL6VXo8EdE0k8ATEQ2yzeDg6m6fxpgSeCL6VCaXI6JxCTwR0bgEnohoVFnyIimMI6JhvRx4Wno6vXrqNCKmkV7Oq5UeT0Sf6uU5nrbW45G0kaSvS7pF0vnVaxtLulbSjZIuqV47VdLB1f4pkg4b7b1jtLF4aAnGTj5YxMxW5nha2bqh3YXAFlPSEO8NbCvp9cC2wPmUtTrmS9oa+Hr1NcA+wLfHeO9vSQrjiM65x1MYtzvU2glYWK3PsTmwHfAT4DjgGOBVwGzbD0raTtKmwNO2n5U02nv/qZ6PEREj9fJQq93A81PgLtuXSHoX8Cjwp5QezlWsnbDvLuAk4JvjvDcipoRxl/Kit6LdwPNF4BJJx1BWnP8TSpriC4DjqzLbUdZl/TrwfWD7cd4bEVPETPPAM7Scoe1ngcNGHL4FeN0o7/kxZeg19PVo742IKdJPQ62ImAacFQgjonndu2LVigSeiD6V9XgionHp8UREs1rLEto1CTwRfchkzeWI6IJeXhYjgSdqJamxtpqew2jys3UuV7UiogvqyJ0+VRJ4IvpQmVtO4ImIRmWoFRHdkMATEU3L5fSIaFyGWhHRqOROj4iuSI8nIhqXwBMRjasr8EhaCrwWuNb2maMc3wy4khJPngH+yPZvxquz3fQ2ETEtGDzY2jYOSYcAs2wvBOZJ2nGUYkcAn7F9APA4sGiis0uPJ6IP2TDY+p3Lc0ck0Fxie0m1P0DJIANwI7AX8LO12/IFw77cEnhiogYTeCL6VBtDrZXjJNCcAzxW7a8CdhirEklvArawfcdEDSbwRPQl1/Ws1jPA7Gp/Y8aYnpH0KuDzwKGtVNpzczzJnR5Rj5pSGK+gDK8AdqPkzFuLpPUpw7GP2X6klXPrucCT3OkR9agp8FwNHCXpM5S8ePdLGnll60+BNwCnSVom6Y8mqjRDrYg+VFdeLdurJA0ABwDn2n4cuHdEmQuBC9upt/Eej6SdR4mYEVErY69uaZuwJvsp21dVQacWjfd4qtTGpzfdbsRMkzuXI6JxCTwR0bCsQBgRDcuayxHRFenxRETDjJPeJiKaljWXI6JxmeOJiEbVdefyVEngiWmr6VzmTf0iL1hQx2OKuZweEV2Q3OkR0bjM8UREs8okT7fPYkwJPBF9yORyekR0QSaXI6JxmeOJiIY5V7Uiolm5gTAiuqKXA09bay5L2lrSfpNtTNKhVSqMiJhS9aQwniotBx5J2wDnAQdL+kmVxmKZpN0lbSPpekm3Szq3Kr+tpOsk3SbpnKqaB4ALJG1Q/0eJiOHc4n/d0FLgkTQPOBc4EfgP4CzbA9V2D/AJ4O+BhcBekhYCHwKWVsned5e0je37KcHr/ASfiKlVU16tKTFh4JG0HXA2cILtX41RbA/gFpdPcRvwRkq+5SMlbWd70VBqDNsPAJ+mBJ8N6/gQEbE22wwOrm5p64ZWejwDwN22Vw177bRhQ61ZwCbAs9Wx54BNgb8GvgPcJOnUEXU+BLwI7DyysaQwjqjHtO7x2P4KsErSicNeHj7UWg2soiR0B5hTff06YCkl3/LbJe0DUAWqzwGX2757lPaSwjiiBtM68ADYvhRYKemjYxS5ExhQWSDlzcBdlKR9b7L9PPAgsKGk9YDzgS/bvr3Tk4+IsfVy4Gn5Ph7bV0g6hDLfY0nHVYcuBM4ALgeOAG61fbukXwNLJL0M/By4HjgNuKiakI6IqdTD9/G0dQOh7W8A3xjj8P4jyt5Huco13J+3015ETI5tBlvIi94tuXM5ok/18p3LCTwRfSqBJyIalsXeI6ILsh5PRDQqy2JERBe4p3s8bS2LERHThz3Y0jYRSUurVSZO76TMcAk8EX2qjjuXq5uGZ1WrTMyTtONkyozU60OtlcAjk3jf3Oq9TUhb06+9SbU1yZTJk2lr+8k0NMJ1tue2WHbDEQ9lL7G9pNofAK6q9m8E9gJ+NuL9rZRZS08HHttbTuZ9kpY39ZBp2pp+7fVrW8PZXlRTVXMoS9xAefh7h0mWWUuGWhExnmeA2dX+xoweM1ops5YEnogYzwrK0AnKEjcPT7LMWnp6qNWBJRMXSVs91FbT7fVrW1PhauDWavnjA4HDJZ1p+/Rxyuw5UaXq5ZuMIqL7JG0BHEBZ3vjxyZZZq3wCT0Q0LXM8EdG4BJ6IaFwCT0Q0LoEnIhqXwBMRjfv/a+NHsM7hF1IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "55m 54s (- 118m 47s) (1600 32%) 0.0417\n",
      "59m 24s (- 115m 19s) (1700 34%) 0.0338\n",
      "62m 51s (- 111m 44s) (1800 36%) 0.0362\n",
      "66m 18s (- 108m 11s) (1900 38%) 0.0293\n",
      "69m 48s (- 104m 42s) (2000 40%) 0.0265\n",
      "> 闭 上 你 的 眼睛 。\n",
      "= close your eyes .\n",
      "< close your eyes . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAADvCAYAAAA0EPUCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFsdJREFUeJzt3X+wXGV9x/H3JzEQSEDoBAhBLW3JqIAGNSrGAHcsSKj9YbFFbNWKtRmngLXTqW01/WELqAw6dBR/XA2i0MGh1dLWWpUWMyDBQC6VmlRA7QQrNoOhaAwULNlP/zhnbw639+7de+/e3WevnxdzJmd3n33Od5fd7332e55zjmwTERFlWzToACIiYnpJ1hERQyDJOiJiCCRZR0QMgSTriIghkGQdETEEkqwjIobAgk7WklZIOqtef4okDTqmiSRtkXTQoONok7Ra0lJJh0paOeh42iRdKumn6vWDJH26z9s/uv5/tUXSeyX9rqSv1rfHJL1rkuccIWm5pH+r/32qpCX9jHuQJB0j6WVzeP6rSvpuzFT9+m/t8PgSSZ+VtFXSG6frb0Ena8DAxyU9C7gCuKXxhXuwvn/QjrT9o0EH0fAB4KeAc4A/HnAsTS8A7q/XzwIelfSseunHF1rALtsjwLOBx6neq7cC7wUel/RaSd+SdJ+kC4F/Aj4LPL3+9/PA8/oQ68DVf+ivAF4p6euN790pklZKuknS7ZIur9sfK+kLdeJ6d93NPcAHJR08qNcxW5KOBD4BLOvQ7GJgu+11wM9LOqxTn0/pYXzFsf2QpPOB+22/VdKZwL/ZflDSp6i+cIM2fghpPfJfbPuJQQQi6TnAbttfl/ReYJWkLY0mr7f97T7HtAh4CLgbuEvSZ4FTgL3AHwIvAs6l+mLPp/3AyyX9M/AY0ALWA08DVgHfpfo8XQs8Wi/vBY4FVgKfBvbZvmOe4xw4SauAd1Mlo7cCl9q+rvH4R4C/BT4E3CZpHfCLwGbbN0j6vKSVtndKugK4StKFtuf9+7phwwbv2bOnq7ZjY2M7qT4LbaO2R+v1/cCrgb/r0MUI1WcYYCuwFvjSVI0XbLKWdBzwUds/17j79cB7gAfr20Ucay/pK/XqImA78NsDiGEJcDXwj5LWU71HrwRWAA8DX+p3ogaw3ZJ0l+2flTRC9d78F/B7tvdKugro1y+TL9p+Q/0HA+AB4JtUX8yDqT5PS+t/XwS8ENgHLAFeBSyW9HTbf96nePuu/t5dClxU//+ZrNmLgffbtqStVO/TA8BrJd1me0O7oe17JL2HKmFfZPuxyTrslT179nDnnXd21XbRokWP2V472WO29wJMU3ldRvW6oRp8HNOp8YJN1lSjnFb9ZV5D9WV6DnCypM8NNLIJbJ866BiA1wHtIcWbgT+gGhVeAvwOB0YAg7CmHuEfAdxINXq9jmo0djBPHt3Mpw11HPvq298GLuDAe3Uo1Qh/H1XJ5mrgZxrPf8j2TX2KdVBGgLvayar2Dklvqtd/FjgMeKS+/ShwONXn7AngS5KusX1Z4/nfpPo+nwjcNY+xA9Dq3/mS9gGHAD8AlnPgczWphV6zxvaFttcDY1Q/VX/N9qYBh1Wia6h+dWD7tcBJwNHUP/+Brw0sMvhqXSt+K4DtbwFPSHou1Ye9X8n683UcbfdSJednUf0xO7j+95314xcARzXa9+1zp8rJ/d6pbvuvgL2SLm7cfantkXrZTzWKXF4/tqy+fTKwmWpgdbakMwAkLQauBK6zPe+J2tVr6GrpgTGqUhpUr3tXp8YLPlkDSDqP6ifGLYOOpVS2W1QjGyQdQZW426OffcBfDii0qVwE7KQazf6wD9tbwoGRdXOH5keofva/gmqU+In6Pqjq2k8Djq+Xfu6LeAvwzPrfvrJ9DbBH0tumaLINGKn/kLwUuIPqD9lLbP8PcB+wtC7NXQVca/v2+Y8cwF3/NxOSXibpogl3fwJ4p6S/pPrVsK1THws9WS+S9HbgbcBvDjoYGJ9CuLjD44slDbo8dTXVCHEvgO3PAk+td9AOwvPqJHll+w7b361HaSts/28fYtgPXFmPrC+iGkW/i2pkfzXVzIfFwOnAmRwoMZ4BbKiXfn7fllDtb/hBH7c5zvb1VOWL86nKIO3ZIK8G/gz4JeB24NY6Ef8F8C5Jt9Sx30T1Gfyw7e39Cxz2t9zV0lV39S8x2zfb/sCEx+6nmtl0G3Bm/Xme0qCTwnwS1cjmXqovzEupdvq0fzI/pX68394AbJTU3vZjjR2MUH2h309Vl+23RVTv2x/Y/oak1Y3H3gh8v98B1X/Yxmy/XNJLgNPq+3+DajT2lU7P7xXbD1LNcMD2fZLOofp5vw3YJukNVDsXr6X6eXso1d79823vknQ21ayVvrB9haSj67gHwvZngM9M8fCZE9ruANZNaPMX8xFXJ6avNWtsfxe4oZu2ysUHIuafpKcMakpmdO/5L3iBb926tau2y5cuHZtqNsh8WMgj64hiJFEPj1IHsEnWERE1230tg8zEQt/BOE7SxkHHMJVSY0tcM1NqXFBubCXG1cepezPyY5OsgeI+FA2lxpa4ZqbUuKDc2IqKy8B+u6ul31IGiYhoSM16FlasWOHjjz++J3094xnPYO3atb057GhsrBfdPImkIj8hiWtmSo0Lyo2th3HtsX3U9M06K7VmXXSyPv7449m+vX/z4bvV5yN4I6I790/fZBoDqkd3o+hkHRHRT+1zg5QoyToiomF/axAHNk8vyToiYtzMT9LUL0nWERE1G7o8R1PfJVlHRDSkZh0RMQSSrCMiCtfvU6TORJJ1RESbndkgERHDIGWQiIjCGTJ1LyJiGGTqXkTEEEgZJCJiCCRZR0QUzgXPBpn1lWIkbelhHBERRSj1sl4ZWUdE1Ib6oBhJS4FrgKcB3wfOs/3ohDYH121WAd8BLgAWA38NHA7sAc4DDgI+CRwNfM32hT16HRERPVHq1L1uyiAbgbttrwc+DZw8SZvfAnbYPgO4D3gjcCLQsn06MAosr/vaUd93rKTnTuxI0kZJ2yVt/973vjerFxURMVstd7f0WzfJ+lnAHfX6NcCdk7Q5EdhWr28Dng3cBeyQ9EXgF4BHgWcCv1zXu38aOG5iR7ZHba+1vfaoo+Z8ObWIiK7ZptVqdbX0WzfJ+h7ghfX624E3TdJmJ3BqvX5qfXsNcJvtlwNHAqcB9wJX2h4BNgHfnnXkERHzoGV3tfRbN8l6FHh+PRp+PnDtJG0+Bpwk6RZgNdUIfBfwFklbgZXAduCjwDl1uzcD/znH+CMiempoZ4PYfoxq5+DE+0ca648Dr5nQ5EfA2ZN0+f/6iogoRQ6KiYgonAdU4uhGknVEREOpU/eSrCMiagb2F3ravSTriIiGUmvWsz43SETEQtSrqXuSNkvaKmnTFI8fKelzkm6V9OHp+kuyjoho63La3nSjb0nnAottrwNWSVo9SbPXAdfZPg04TNLaTn0mWUdE1MyM5lmvaJ8ao142NroaAW6o128G1k+yuYeAZ0o6Ang60xwkmJp1RETDDKbu7bE91Wh4GfBAvb4XOGGSNl8GXgG8hepI8Yc7bSwj64iIhh7VrPcBh9Try5k8114GvNn2n1Ml6ws6dZhkHRFRa5/PugfJeowDpY81VKffmOhQ4DmSFgMvrjc/pSTriIi2Hu1gBG4EXifpfVSn2Ngp6ZIJbd5Fde6lHwA/AVzfqcPUrCMiGnpxuLntvZJGgLOAy23vBu6e0OYO4KRu+0yyjoiotWeD9KQv+2EOzAiZs6KT9djYGJIGHUb0QKlHheXzFROVenXzopN1RER/OSdyiogonV0tJUqyjohoyPmsIyKGQKn7V5KsIyJq7YNiSpRkHRHRZtPKbJCIiCGQkXVERPmcy3pFRJSv0IF1knVERFs1z7rMbJ1kHRHRkGQdEVE809qf2SAREUVLGSQiYkgkWUdEDIMk64iI8hWaq5OsIyLGOTsYIyKK18vLevVaknVERMOCStaS3gncY/t6SX8K7AI2AKuA7wAXAG8HttjeIukNALavkbQFuBN4ru2zJ+l7I7BxNnFFRMxVqcl60Syf90ngNfX6BuAwYIftM4D7gDd2eO6pwO2TJWoA26O219peO8vYIiJmx4ZWl0ufzSpZ2/4WcJikEWAHcCKwrX54G/DsCU85pLG+w/ZnZrPdiIj5Zrurpd9mO7IG+BRwNdUoeyfViJn6353Aj6hG3FCNvtv2zWGbERHzxkCr5a6WfptLsv4bqtf2ZeBjwEmSbgFWA9cAfw/8vqQPAw/NMc6IiPnnckfWs93BeBLwceAyV1E/zoEadtsO4PSJz7U9MpttRkT0w4K6+IDtncCLehxLRMSADWbU3I3Ms46IaEiyjogoXE6RGhExJLw/yToiongZWUdElG5A0/K6kWQdEdGQZB0RUbicIjUiYhgYXOjFB+ZyuHlExALT3aHm3Yy+JW2WtFXSpmnafVDSL0zXX5J1RERDNdd6+qUTSecCi22vA1ZJWj1Fu9OAlbb/Ybq4kqwjIhpmMLJeIWl7Y2leNGUEuKFevxlYP3E7kpYAHwV2Sfql6eJKzToiombP6EROezpcJGUZ8EC9vhc4YZI2rwf+HbgcuFjSM2y/f6qNJVlHX0gadAiTKnXPf6nv14+DHn0m9nHgoivLmbyK8Txg1PZuSdcBlwJTJuuUQSIixplWq9XVMo0xDpQ+1lBdp3aibwI/Xa+vBe7v1GFG1hERbb07kdONwK2SVgHnAOdLusR2c2bIZuBqSecDS4Bf6dRhknVERFMPLj5ge299jdqzgMtt7wbuntDmh8CvdttnknVERK06grFHfdkPc2BGyJwlWUdENJS60znJOiKizaZV6OHmSdYREQ0ZWUdEFC5n3YuIGAa93MPYY0nWERHjcqWYiIih4DL3LyZZR0SMM90cSj4QSdYREbXsYIyIGBJJ1hERxfNMzmfdV0nWERFtvTvrXs8lWUdENCVZR0SUzUBrIZVBJB0KfBI4GvgasAe4x/b1kv4UuBf4+2Yb2xdKOgT4a+Dw+jnn2X5i7i8jIqIHZnYNxr6a7WW9NgI7bJ8OHAt8BnhN/dgGqqskPKmNpOcCJwKt+r5RqmuTPYmkje2rBc8ytoiIWeruyuaDqGvPtgzyTGBdfSWEI4BVwGH17R22H5M0sc1xwOeBHZK+CHyD6hLtT2J7lCqRI6nMP3ERsWCVuoNxtiPre4ErbY8Am4BvA58CrqYqfUzVZg1wm+2XA0cCp8068oiIeVDqyHq2yfqjwDmSbgHeDPwn8DdU9fkvd2izC3iLpK3ASiCljogohg3e3+pq6bdZlUFsPwKc174t6STg48Blrv/kTGzTcPZsthkR0Q+FVkF6M3XP9k7gRb3oKyJicHKK1IiIoZBkHRFRuhxuHhFRPlPuQTFJ1hER44xz8YGIiMKlDBIRMRwKzdVJ1hERTalZR0QULtdgjIgYBqlZR0QMA9PKbJCIiPKlZh0RUbqqaD3oKCaVZB0/1iQNOoRJlVo3hXLfs14oOFfP+nzWERELUq8uPiBps6StkjZN0+4YSf86XX9J1hERbTat/a2ulk4knQsstr0OWCVpdYfmVwCHTBdaknVERMMMRtYr2hf3rpeNjW5GgBvq9ZuB9ZNtS9LLgEeA3dPFlZp1RERthgfF7LG9dorHlgEP1Ot7gRMmNpB0EPAnwCuBG6fbWJJ1RERDj3bu7uNAaWM5k1cx/hC4yvb3u9lpmzJIRMQ411fN7WLpbIwDpY81VBcLn+hM4EJJW4BTJH2sU4cZWUdEtBncmwMYbwRulbQKOAc4X9Iltsdnhtg+vb0uaYvtN3XqMMk6IqKhF4eb294raQQ4C7jc9m7g7g7tR6brM8k6IqLWy7Pu2X6YAzNC5izJOiKiLWfdi4gYBs6JnCIihkJG1hER5TNJ1hERRbNNq7V/0GFMKsk6IqIhOxgjIoZAknVExBBIso6IKFx1+tNcMDcionhJ1l2qT+C9cdqGERHzIGWQLtkeBUYBJJX5rkXEgpVkHRFRvHJr1n2/+ICkEyVd0u/tRkRMx+7d1c17re8ja9v/DnS8NHtExKCkDBIRUTzjHlx8YD4kWUdENJgk64iI4qUMEhFROOdKMRERw2AwMz26kWQdEdGQ81lHRAyBjKwjIkpXFa0HHcWkkqwjImom12CMiBgKpZ4bJMk6ImJcZoNERAyFVg43j4goW7V/Mck6IqJwKYNExAxIGnQIUyo1mfXsPSv09SVZR0Q0ZOpeRMQQKPWXQ5J1RETNds4NEhExDDKyjogYAknWERFDoFfJWtJm4NnA52xfMsnjTwU+RZWH9wGvtv2jqfpb1JOoIiIWBINb3S0dSDoXWGx7HbBK0upJmv068D7bZwG7gQ2d+szIOiKiZkOr+yMYV0ja3rg9anu0Xh8BbqjXbwbWA9948rb8wcbNo4AHO20syToiomEGZZA9ttdO8dgy4IF6fS9wwlSdSHoJcKTtr3TaWJJ1RMQ49+rcIPuAQ+r15UxRcpb0E8D7gVdN12Fq1hERDba7WqYxRlX6AFgD7JrYQNJBVKWSP7J9/3QdJllHRDT0KFnfCLxO0vuA84CdkibOCPlN4AXAOyRtkfTqTh2mDBIRUatOkTr3qXu290oaAc4CLre9G7h7QpsPAR/qts8k64iIccbuzeHmth/mwIyQOUuyjohoKPUIxhnVrCUdI+lls92YpFfVRfWIiCL1qGbdc10na0krgSuAV0r6el0Q3yLpFEkrJd0k6XZJl9ftj5X0BUlbJb277uYe4IOSDu79S4mImKvuEnWxyVrSKuBy4GLgv4FLbY/Uy1eBdwJ/C6wD1ktaB/wOsLk+3PIUSStt76RK+FclYUdEadrXYOxm6bdpk7Wk44DLgItsf3+KZi8GbnH152Yr8EKqo3deK+k42xvqvaHYvgd4D1XCXjrJ9jZK2j7hMM6IiL4Y5pH1CHCX7b2N+97RKIMsBg4DHqkfexQ4HPgA8E/AlyS9fUKf3wQeB06cuDHbo7bXdjiMMyJinhi3Wl0t/TZtsrb9V8BeSRc37m6WQfZTHfu+vH5sWX37ZGAz1dE7Z0s6A6BO7lcC19m+q3cvJSJi7tzlf/3WVc3a9jXAHklvm6LJNmBE1eWFXwrcAWwCXmL7f4D7gKWSlgBXAdfavn2uwUdE9FqpNeuu51nbvr4+R+tlgCW9qX7oQ8CfAddRnZ/1Vtu3S/ohMCrpCeA/gJuAdwAfrndKRkQUpVdHMM4HlRoYgKRyg4v4MVVqzpA0Ntd9XQcdtNTHHHN8V22/851757y9mcgRjBERDa0B7DzsRpJ1RETDIOrR3Uiyjohoq4rWg45iUknWERE1w0Cm5XUjyToioqHUHahJ1hERDalZR0QUz5kNEhFRupIPikmyjohoSLKOiCieITXriIjyZepeRMQQSBkkIqJwtmm19g86jEmVnqz3APf3qK8VdX8lKjW2xDUzpcYFPYytOm19z/TyPfvJXnSSkfUs2D6qV31J2l7qpcJKjS1xzUypcUG5sZUYV5J1RMQQSLKOiBgGSdYDNzroADooNbbENTOlxgXlxlZUXLZpucwdjEVf1isiop8WLVrsgw8+pKu2jz32SC7rFRExKKUOYJOsIyLGOck6ImIY5HzWERGFyylSIyKGgosdWS8adAARESWxW10t05G0WdJWSZvm0qYtyToiosF2V0snks4FFtteB6yStHo2bZpSBomIOOALtld02XappO2N26O22wf5jAA31Os3A+uBb0x4fjdtxiVZR0TUbG/oUVfLgAfq9b3ACbNsMy5lkIiI3tsHtA+FXM7kubabNuOSrCMiem+MqqwBsAbYNcs243JukIiIHpN0OHAr8C/AOcD5wK/a3tShzam2fzBln0nWERG9J+lI4CzgFtu7Z9tmvG2SdURE+VKzjogYAknWERFDIMk6ImIIJFlHRAyBJOuIiCHwf9217nqI9cjJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "73m 19s (- 101m 16s) (2100 42%) 0.0300\n",
      "76m 45s (- 97m 41s) (2200 44%) 0.0267\n",
      "80m 11s (- 94m 8s) (2300 46%) 0.0250\n",
      "83m 40s (- 90m 38s) (2400 48%) 0.0237\n",
      "87m 11s (- 87m 11s) (2500 50%) 0.0246\n",
      "> 汤姆 喜欢 热 的 。\n",
      "= tom likes it hot .\n",
      "< tom likes it hot . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAD8CAYAAAAMs9NCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFwtJREFUeJzt3X+cXXV95/HXOwEEEkQ0SAi2UB/wsPzQpNv4AGOQqRAJ1bU2VkRbtv6gWWuBurvWBUld7CLY6LI81qLdqdlNV1y6dFtZ61pbaoikBJEMRgUK6rrgLjUlETBEoC657/3jnIHLMHPnTs6Ze8+cvJ95nMece+93vuc7kzuf+z3fn7JNRERbzRt2ASIiZlOCXES0WoJcRLRaglxEtFqCXES0WoJcRLRaK4KcpKMkHVaeH9Aj3XsHV6qZkfQGSUcMuxwRbdOKIAecBHy+DBKfk3SLpM2Sdkq6E0DS84DV5flqSfeVaTZLeqR8vTJJyyV9WdIXyuNbkr7e9XiTpDMknSLpZ8tjPvAR4Ik6yjCDsn6+6/xtw/4QkPQRST9Tnh8k6c+GWZ5hKT+0X1vh+98s6aA6yzSXqS2DgSWdA3zN9g/Lx28C3g28HTgAOBM4FbgWeCXwPODFwL8H/sb2mTWW5eXAzwAd4BxgD7CF4kPlm+XXtcBTwLll2e4C7iizMPAW24/WVaYJ5XsFcDVwInAvcD1wAaAyyWkewhtD0peAX7TdkfR64DyK4A/wPds/GXSZBk3SYuBjwCPAKuAfypfeB+wAPgMsBLbY/oCko4GNwGHALbYvkXQy8C+A37L9jwP+ERpnztfkJP2UpGuBv+0KcL8BvBFYY/sxYAnwHuAhijfJYcCjwN8DR81CsQ4rr/kS4FvA/y7PXwIssP094Cu21wH3UNQwLwbOBl4PHDBbAQ7A9jdtn0UReH8F+FPgU8DvAj8YdICTNE/SI8DBwJ2SrgB+E5gPXAL8OfDSQZZpGCQtAdYDFwEPAx+xPVIe24EPA58DVgArJa0AfhvYYHsFsEzSYtt3Ax8Hrq3rDmUum7L9ag7ZAeykqJFdIGkLcBrwNeBmSQBXUQS1/0ARRD5IUbt6lCL4LZV0le1LayrTfcAXKGpt3V4OHFuer5Y0QhHkXgpcCTwA/CzFH/askXQKRc1gBfB7wBjwUeB1wMA/+cua2522zyx/J+8FfgD8K9u7yw+xVtfiJB1DUWu9sPyZJ0t2KvAJ25a0leKO5EHg1yTdanv1eELb90r6fYpAd6HtJwfwYzTSnA9ytv8fcLmk8VrpWopPsfUU1f2HgOOBH1EEv5soagifA060fYOkvwbW1Visn1DUPg4Dbiufew1FjWlv+fh3KALc+23fKOlg4EDbe4DbayzLZL4PXE5Rq7xI0lHAuyg+KJ6QdCrwIttfnOVydFsqaTPwAuBGihr3dRQ18ucBbf8jHQHutL2767nLJF1Qnp9J8X76cfn4ceD5wBUUzR43S9po+8qu7/8uxYfWScCds1j2Rpvzt6sAkn4OuFLS84FPUNx2vYeirelFwHaKIPfvgP8JHAHsAk4se2Uft713srz3oSz/miKAHgf8EvCG8lhNEWz/QtJlZbl+ALxPxcf2GeX3r6ijHL2Uf0i/DCwsO2v+hKIN51yKtro/pbhdGqTttkcoapjY/l/AU2X74SEMOMipcIqmqFLVzfZngd2SLup6uvt2dS+wm6I9DmBB+fgUYAOwFDhb0vj7aD5wDXCd7f02wEFLghxFG8ZtFDWADwM/DTxZtk0cSXFL+2ngUOBtFB0O/xU4Hfg8cJqkd9ZUlqsp2rm+RFE7/FJ5fKj8ejZFLfMa4FKKmuUryvPVwMclvaCmsvSyEzifIuC/nyKIdIDHgN+0/dUBlGE6FwJ3U/y/PTbga18MvKz8OhC2NwK7JH1giiS3AyNl4H01RZPMOuBVtp8Avg0cLOlAig62z9i+bYq89h+25/RBUWO6l6Jn8ACKBuvNFJ/+hwO3UnzK3QIcDby963vnUdxGfoyisb+uMh0D3Awsm3B8syznzwOfLtMupgjO7y4fvx74OjAyi7+zwyn+YA4E/htFO+HbKW6hF5a/s4MG/P/4w/L/bTtw+YTXtgzhffV+4J8D7xjCtdeU7+m/K38nm4G3lu+VvwG+CnysTHsKsLV8f28s39O/CywbdLmbesz5ISSSXgmcYvs/SzoR+A3gg7afLBtef0RRy/sH2/dM+N4TKW5h/6Xte2ss07EUjfl3TXjpnwAvtP1UV9pXA4tt/1nXcy8DOra/U1eZJpTvF4DDXbQFrqSoxX2LYtjKdZLeCNxh+wezcf1JyjMf+Evbr5P0KuB02+sl/TpFTeWrts8fRFkmlOvFth8a9HWjXnM+yDVR2QmywMXwldhHZXvhAbZ3DrssMXclyEVEq7Wl4yEiYlL7RZCTtHbYZZioaWVKeXprWnmgmWVqov0iyFEMEG6appUp5emtaeWBZpapsnKBgi09Xj+wXOxiq6R3TZff/hLkImIOKDub/phisPNULgK2uZiv+4ZyQP/UeTa542HRokU+7rjjKuezc+dOjjzyyMr5jI2NVc4jooF22d7nP5DVq1d7165dfaUdGxu7m2fPXhm1PTr+oJy1JOB/uJgB8xzlEmGX2L5H0vuBMds3T3XNRs9dPe6449i2bduwi/G0Ac3wiRi0B6p8865du7jjjjumTwjMmzfvSdvLp3rd5dzdaf7WFlAsTADF1LaeKwk1OshFxNzQGewd4R6KGU0/opihs6dX4rTJRUQlpv/poTUZA1aW50uB+3slTk0uIioyZnZqcuUy8CfZ/oOup/8Y+KKk0ymWkeq5NFmCXERUY9jbqTfIjXc62N4EbJrw2gOSVlHU5j7kaZZJS5CLiErMwNvksP33wA39pE2Qi4jKmjwULUEuIipLkIuI1rI98NvVmUiQi4jKUpOLiNYysDdBLiLaLDW5iGi1tMlFRHvVO2WrdrUGOUnLAGxvrzPfiGiu8bmrTVV3TW5Z+TVBLmI/srfTGXYRplRbkJN0FfDL5fn5wC9SbHa7BPi/wDsp9j99CPgJxUa519u+pq4yRMQwzN4E/TrUttSS7UuBjwIftX0mxSbPd9k+A/g28C7gUOAtwCsodmxfOjEfSWslbZO0befObLcZ0XQ2dPo8hmE215PrXgLlduBEil3s91CsRLqXYpnjZ7E9anu57eV1LFkeEbNvwOvJzUjdQe4JitoawN3AaeX5aeXjiGih/SnI3QSskXQrcBdwsqRbgBMo2uciomXGl1rq5xiGWntXbT8MnNX11FcmJBkp042Uj99R5/UjYgjs/aN3NSL2X/vTOLmI2M8YGj2EJEEuIiob1vCQfiTIRURluV2NiFZLkIuI1nJ6VyOi7VKTi4jWGsa+qzORIBcRlWUISUS0WoaQRERr2aaTjoeIaLO0ye2jsbExpOcsOTc0TexBatLvJ/ZfTfzbGNfoIBcRc0OCXES0loe4Vlw/EuQiorIMIYmI1jKwt8FjSBLkIqKyJrfJzeZuXRGxn6hzjwdJGyRtlbRuitePkPRFSVsk/eF0+SXIRUQ1fe7U1U9tT9IaYL7tFcASSSdMkux84DrbpwOHSVreK88EuYioxMxoS8JF45vHl8faCdmNADeU55uAlZNc8ofAyyS9APgp4Pu9ypc2uYiobAZDSHbZ7lXzWgA8WJ7vBo6fJM3fAq8HLgbuBR7pdcHU5CKishrb5PYAh5TnC5k8Rl0JvMf271EEuXf2yjBBLiIqqXlz6TGeuUVdCtw/SZpDgZdLmg+cWhZhSglyEVFNjR0PwI3A+ZKuBs4F7pZ0xYQ0VwGjwI+AFwLX98owbXIRUVld07ps75Y0AqwC1tveAXxjQpqvASf3m2eCXERUMt67Wlt+9iM808NaWYJcRFSW3boiosXc6An6fXU8SNo84fE1XefvkPSOeosVEXOF3f8xDPtUk7P9vroLEhFzV5PXk9unISQTa3blcydL2iTpMEmHSvrvkm6RdG35+iGSvlA+9+eSJg2wktaOT/nYl7JFxODVOISkdnWNkzsa+CzwdtuPAWuBu2y/Bjha0iuAk4BO+dwoxWjm57A9anv5NFM/IqIhah4MXLu6Oh4uBL4OHAvsAF4GrCjHu7wAOAb4EnCXpL8GvkMx+TYi5rqGb0lYV03u3wLvLb8C3AdcY3sEWEexSsBS4FbbrwOOAE6v6doRMWwN7nmoK8g9afv7wL2S3gj8EXCOpFuA9wD/h2IO2sWStgKLgbS5RbSEO+7rGIa+blfLGtmkj21v7Dq/uCvZuZNkdfaMShcRc0KDO1czGDgiqinuRJsb5RLkIqKyBLmIaDHT2dvc3tUEuYioJLerEdF6CXIR0W4JchHRZg2OcQlyEVGR0/EQES1W9/LndUuQi4jKEuQiotUS5CKivWwY0uT7fiTIzYCkYRfhOR7es2fYRXiWl7x4ybCL8CyPP7572EXYL6QmFxGtZaCTmlxEtFamdUVE2w1rQcx+JMhFREXD24mrHwlyEVFZglxEtFaWWoqI1vPeBLmIaLHU5CKivZyOh4houQS5iGitLLUUEe1mcIMXzZw37AJExFxXtMn1c/RD0gZJWyWtmybdJyX90+nyS5CLiMqKsXLTH9ORtAaYb3sFsETSCVOkOx1YbPsvpsszQS4iKptBTW6RpG1dx9oJWY0AN5Tnm4CVE68l6UDgj4D7Jf3SdGWb9TY5SdfYfl/X42UAtrfP9rUjYvbZM5qgv8v28h6vLwAeLM93A8dPkuafAfcA64GLJP207U9MleGs1+S6A1xpWXlEREvU2Ca3BzikPF/I5DHq54BR2zuA64Bf6JXhrAc5SZu7zq8CLgEukfTlKdKvHa/KznbZIqIOptPp9HX0YYxnblGXAvdPkua7wEvL8+XAA70yHOgQEtuXSrqvPN84RZpRYBRAUnMH30REod4J+jcCWyQtAc4BzpN0he3untYNwH+SdB5wIPArvTLMOLmIqK6mRTNt75Y0AqwC1pe3pN+YkOYx4C395jmMIPcE8CIASXKTh0pHxLSKGQ815mc/wjM9rJUNYwjJTcAaSbcCpw/h+hFRszoHA9dt1mtytkcmPH4YOGu2rxsRA2LTafC0rrTJRURlTW51SpCLiEqyCklEtFvdPQ81S5CLiIqyMnBEtJyb2++QIBcRFZl+p2wNRYJcRFSSjoeIaL0EuYhoMc9kPbmBS5CLiGrqXYWkdglyEVFdglxEtJWBTm5XY7YcsWDBsIvwLI8/vnvYRYhBm9keDwOXIBcRFWXGQ0S0XIJcRLRaglxEtJYNzqKZEdFmDa7IJchFRFXpeIiIlkuQi4j2yrSuiGgzk8HAEdFqxlk0MyJaK7erEdF2DY5xCXIRUV3a5CKitbLHQ0S0W8Pb5ObVlZGkyyWN9JFumaRldV03IobNdDqdvo5hGEZNbjzAbR/CtSNiFjS5Ta62mlxplaSvSNou6VhJ15ePPyvpIElXAZcAl0j6cs3XjohhKBrl+juGoO6a3PG2z5D0AeDXgbtsv03SvwHeZftSSfcB2N44WQaS1gJray5XRMyS8RjXVHXX5P5L+fUhihrb7eXj24ET+8nA9qjt5baX11y2iJgltvs6+iFpg6StktZNk+4oSV+fLr+6g9yPu85/BzitPD8NuLs8fwI4FECSar5+RAyaTWdvp69jOpLWAPNtrwCWSDqhR/KPA4dMl2fdQa7bU8DJkm4BTgA2ls/fBKyRdCtw+ixePyIGZAY1uUWStnUdE5umRoAbyvNNwMrJrifptRSVqh3Tla22Njnbl3edbyxP/+Mk6R4GzqrruhExXDMcDLxrmqaoBcCD5flu4PiJCSQdBHwIeBNw43QXzGDgiKisxsHAe3jmFnQhk99tXgJca/vRflq8ZvN2NSL2C30OH+kvEI7xzC3qUuD+SdKcBfyWpM3AMkmf7pVhanIRUY3B9U1muBHYImkJcA5wnqQrbD/d02r7NePnkjbbvqBXhglyEVFZXVO2bO8up4euAtbb3gF8o0f6kenyTJCLiErqXoXE9iM808NaWYJcRFTT8FVIEuQioiI3eoJ+glxEVJeaXES0mUmQi4iWsk2ns3fYxZhSglxEVJaOh4hotQS5mDVZraq3Jv7xtfH/rIm/53EJchFRSbGM0nA2qelHglxEVJYgFxGtltvViGi1BLmIaLG0yUVEizkT9COi7RLkIqLFjGtaNHM2JMhFRGUmQS4iWiy3qxHRWul4iIiWc4JcRLRb1pOLiFZLTS4i2qtolBt2KaaUIBcRlZjs8RARLZe5qxHRYuldnRFJa4G1wy5HRPSvk2ld/bM9CowCSGrux0NEAOP9DglyEdFazb5dnTfoC0o6SdIVg75uRMyi8WEk0x1DMPCanO17gHWDvm5EzJ4MIYmIVmvy7WqCXERUYjtzVyOi3VKTi4hWS5CLiFarM8hJ2gCcCHzR9nNGYkg6HPgTivi1B3ir7Z9Mld/Ah5BERNsY3OnvmIakNcB82yuAJZJOmCTZrwJX214F7ABW98ozNbmIqMSGTv8zHhZJ2tb1eLSc5TRuBLihPN8ErAS+8+zr+ZNdD48EHup1wQS5iKhsBreru2wv7/H6AuDB8nw3cPxUCSW9CjjC9ld7XTBBLiIqcp1zV/cAh5TnC5miSU3SC4FPAG+eLsO0yUVEZbb7OvowRnGLCrAUuH9iAkkHUdzSXmr7gekyTJCLiMpqDHI3AudLuho4F7h7krnu7wZ+HrhM0mZJb+2VYW5XI6KSOvddtb1b0giwClhvewfwjQlpPgV8qt88E+QioiJj1zety/YjPNPDWlmCXLSapGEX4TmaNjugjt9R036mbglyEVFZglxEtFizVwZOkIuISrLHQ0S0XmpyEdFixtmSMCLaLHs8RESrpU0uIlqrzhkPsyFBLiIqyhCSiGi5TjoeIqLN0iYXEe1VNMoNuxRTSpCLiEpMhpBERMs1ueNhRisDSzpK0mv39WKS3lwuXRwRLWJ3+jqGoe8gJ2kx8HHgTZL+rlx2eLOkZZIWS7pJ0m2S1pfpj5b0V5K2Svpomc29wCclPa/+HyUihsN0Op2+jmHoK8hJWgKsBy4CHgY+YnukPLYDHwY+B6wAVkpaAfw2sKHcJHaZpMW276YIlNcm0EW0w/hg4Jr2eKjdtEFO0jHAlcCFth+dItmpwC0ufoqtwCsp9k78NUnH2F5drtWO7XuB36cIdAdPcr21krZN2IA2IhpsTgc5ih2t77S9u+u5y7puV+cDhwE/Ll97HHg+8AfAXwI3S/rghDy/C/wjcNLEi9ketb18mg1oI6IxDO70dwzBtEHO9meB3ZIu6nq6+3Z1L8VO1wvL1xaUj08BNlDsnXi2pDMAyqB4DXCd7Tvr+1EiYljc579h6KtNzvZGYJekD0yR5HZgRMWOGK8GvgasA15l+wng28DBkg4ErgU+Y/u2qoWPiGZo8u1q3+PkbF8vaQ1F+5wlXVC+9CngcuA64FeBLbZvk/QYMCrpKeB7wE3AZcAflp0VEdECtul06tuSsG5q8iA+Sc0tXMQ+atrfnKSxKm3g8+cf4AULDu8r7WOPPVzpWvsiMx4iorKmBe5uCXIRUVmCXES0W4JcRLSVbTpubsdDglxEVJbb1YhotQS5iGixbGQTES2XPR4iorWy72pEtJwbXZOb0fLnERGTqXP5c0kbyhXF11VJMy5BLiIqq2sVknIRkPnliuJLJJ2wL2m6Nf12dRfwQA35LCrzapKmlSnl6a228hQrktWirjIdW/H7/8r2oj7THjxh1e9R26Ndj0eAG8rzTcBK4DsT8ugnzdMaHeRsH1lHPpK2NW2l4aaVKeXprWnlgeaUyfbqGrNbQLF1AhSL7x6/j2meltvViGiSPcAh5flCJo9R/aR5WoJcRDTJGMXtJxRbJ9y/j2me1ujb1RqNTp9k4JpWppSnt6aVB5pZpqpuBLaU26CeA5wn6Qrb63qkOa1Xho1eGTgi9j+SjgBWUWxzumNf0zydNkEuItosbXIR0WoJchHRaglyEdFqCXIR0WoJchHRav8f4n1tNSbex/sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "90m 38s (- 83m 39s) (2600 52%) 0.0233\n",
      "94m 11s (- 80m 13s) (2700 54%) 0.0292\n",
      "97m 40s (- 76m 44s) (2800 56%) 0.0323\n",
      "101m 13s (- 73m 18s) (2900 57%) 0.0289\n",
      "104m 44s (- 69m 49s) (3000 60%) 0.0240\n",
      "> 我 快 好 了 。\n",
      "= i m about ready .\n",
      "< i m about ready . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAD8CAYAAAAMs9NCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFRhJREFUeJzt3X2wXVV9xvHvQ0ASElQoGAijUgujAgrWVDEGuKgZwiijwoD4OrXSjG8o01p8IbXYQaxIKVaC9mpaRBSMTsVqbS0WESyIJlFaYkGpEh0UJQqGIAjkPv1j7wuXw73nnOTsnLOz8nwye3Je1ln7dzK5v7te9lpbtomIKNVOow4gImJbSpKLiKIlyUVE0ZLkIqJoSXIRUbQkuYgo2g6T5CTtKkmjjiMihkslXycnaRZwBHAQMB/4HnCP7f8YQSw7ATvbvr9+vrPtB4cdR0dMe9r+9ShjqOPYGdjskv8zbiFJ84GDbV+5lZ8/AfjS5P+3HdnOow5gW5C0O3AZsDdwF/AhYE/gPuANkp5k+xNDDuu5wEuAM+rn75J0s+3PDTmOqf5d0tHArsBhwMSU99bYvntIcbwROFlSZ9J/NnC97RcNKY5WkLQP1f/ZOyWtAH5Rv3UacDvwKWAecI3t0yXtC1wE7A5cbftdwE3AhZLeYvt3w/4ObVJqd/Ue4GXA6cC1tq8ANgMPAK8D1tUtq2F6AHhAlV2B+4GNQ47hIZJmAz+1fQ9VknsysD9wJPAZ4LHDisX2BbYX2x6zPQa8guqH+bP14x2GpAXAOcCpwK+B90/+u9j+HvA+4AvAImCxpEXA24GVthcBh0nax/Y64FxgRf3/bYdVZHdV0ouAdwCPA34P+AHVD/B9VD88OwN/b/srQ4rnYOA4qtbSz6i6z08E7gR2Az5k+9JhxFLH82Lgg8BjgHuBcdsrJB0KnAecYvvHw4qnI7ZjgL8DThvFsMIoSdoPeD/wNtsbJZ0J3GL7killvge8xvaNks4FfkrVAl8CvMn2bR11Hgi8E3ir7fuG9FVapcjuqu2vAV+TNAaMUf1AnwjMAlYBHx5WgqsdSZVsbwLm2l4q6R3Aaqrk+8AQY8H2v9YtgCuAo4Ab6reeA3xqVAmu9mTg4h0twdXGgLW2p7bwz5B0Sv34hVRd0nvq57+lanGfBTwIfF3SRbbPnvL5W4DfUf1iXbsNY2+tIrurkhZIeh2wDHgVcCFwM/A0qhbe9UMO6W7g6vrxeklXAKfwcLfkDEkvH3JMRwDfBBYCz5f0Daoxn7+Q9J+S/nzI8Uya6F1k26uHFQ4Z5oy87U8DGyWdOuXlqd3VzVRDHPPq9+bWzw8BVgKHAsdIOqr+DrOA84FLbO+QCQ4KTXLAPlTdwW8Bn7H9eqrfYscBh9j++DCDqbsbP68fX2B7CXAjcBtwpO1n2f7CsOKpZzN/BZwN3GX7g7aPohrs/pDtF9r+22HF01JvA55a/z00ti8CNkg6fYYi1wNjdfJ9PvBtYDnwPNv3Ug3NzJa0C7CCqmV+3baPvL1K7a6uBdbWM4dHSno88JdUkw83SXo68Azbq0YRn6TnUyWZT1L9pv3TYZ7f9oOSTqRK/D+WtOuOPgM3jV2AvYDfDPvEti+VdDzVLyFP6a5+FDgTuAR4NdXs6nWS7gbG69npH1ENQ5wBfKyerNihFTnx0EaSJi8h+QTVdP/Jtn8h6QNUP1Cn2x5KV03SHOAfgK9STXw8xfa7Jb0Z2GT74mHEMU1cbwJeCXzW9opRxNARzxNs/3LUccRgkuQiomiljslFRABJchFRuB0iyUlaNuoYOrUtpsTTXdvigXbG1EY7RJKjul6ubdoWU+Lprm3xQDtjGpik+ZKu6fL+LpK+LOlaSX/Sq74dJclFxHZA0h5Ul1bN7VLsVGB1vVb3JfWGHDPX2ebZVUntDS62E00tWHBjdT3rD5/VSD0b7riDvfbee+B6vrt27QbbA1W0dOlSb9iwoWe5NWvWrKNaQz5p3Pb45BNJj6X6h/5ivVnDo0j6F+Bdtr9fL49cY/vrM52z9RcDD3+zkJkN6TK2aNDOO+8y6hAe5Zpr/2vUITzCvNlz1g9ax4YNG/jOd77Ts9xOO+10n+2FM70/uW63x2q6uVSrhaBa1ja/W+HWJ7mI2D5MDK9XuAmYQ7UaZV79fEbtaSZFxHbLgO2eR0PWAIvrx4cCt3YrnJZcRDTAmOZbcpJeABxk+4IpL38S+IqkyVsbdN1VKEkuIgZn2DzRXJKbnHSo73FxZcd76yUtoWrNvbfegmpGSXIRMTAz1DE5bP+MagPcnpLkIqIRbb0cLUkuIhqRJBcRxbI91O7qlkiSi4hGpCUXEcUysDlJLiJKlpZcRBQtY3IRUa5ml201aiRrVyWdP4rzRsS2MeS1q1tkJC0526eN4rwRse1snmjnVmSjasldNYrzRsS24r7+jELrxuTqm3MUuXd9RKlsaHB9fqNal+TqrZDHIdufR2xP2jrx0LokFxHbpyS5iCjWsLda2hKjml0dG8V5I2IbsVs7u5qWXEQ0It3ViCiWYWSXiPSSJBcRjcglJBFRtHRXI6JoSXIRUSxndjUiSpeWXEQUKxcDR0TxcglJRBQtl5BERLFsM5GJh4goWcbktpLdnt8ObZw9kjTqEFrtwQfvH3UIjzJv9pxRh7BNtPHnA7aDJBcR24ckuYgolu10VyOibLmEJCKKZWBzS68hSZKLiEa0dUxuJPddjYjyTNTjct2OfkhaKelaSctneH8PSV+RdI2kj/WqL0kuIgZn4z6OXiQdD8yyvQhYIOnAaYq9FrjE9hHA7pIWdqszSS4iBmboN8ntJWn1lKPzRvJjwKr68ZXA4mlO9yvgqZIeDzwR+Em32DImFxGN6LM7usF2t5bXXOC2+vFG4IBpynwTeDHwNuAm4M5uJ0xLLiIa0dCY3CZgcknIPKbPUWcDb7T911RJ7vXdKkySi4iBTe4n10CSW8PDXdRDgVunKbMb8AxJs4Dn1qefUZJcRAyuoYkH4HLgtZLOA04C1kk6q6PMB4Bx4DfAnsCl3SrMmFxENKKJZV22N0oaA5YA59i+Hbiho8y3gYP7rTNJLiIGNjm72khd9p08PMM6sCS5iGhE7tYVEQVzFuhHRLns6mijoSc5SWuAXwL3A/sAl9o+f9hxRESz2rqf3CguIdkNOBF4JvAqqmthHiJp2eSSjxHEFhFbqaFLSBo3iu7qL2xvkrQe2Aw84iYFtseproFBUjt/NUTEI+Tm0hFRttySMCKKl5ZcxfbY1L+BPx52DBHRPGf784goWUsbcklyETG46jq5dma5JLmIaESSXEQUzExszuxqRBQq3dWIKF6SXESULUkuIkrW0hyXJBcRDXAmHiKiYE1uf960JLmIaESSXEQULUkuIsplQxboR0TJ0pKLiGIZmEhLLiKKlWVdEVG6bJoZEQUb3d24ekmSi4hGJMlFRLGy1VJEFM+bk+QiomBpyUVEuZyJh4goXJJcRBQrWy1FRNkMbummmTuNOoCIKEE1Jtfr6IeklZKulbS8R7kLJR3Xq74kuYhoRHWtXPejF0nHA7NsLwIWSDpwhnJHAPvY/lKvOpPkIqIRfbbk9pK0esqxrKOaMWBV/fhKYHHneSTtAnwcuFXSS3vFtcVjcpLOBK6yfdWWfrb+/P7A/lv7+YhoH7vvBfobbC/s8v5c4Lb68UbggGnKvA74PnAOcKqkJ9n+yEwVjqIltz9Vto6IgjQ0JrcJmFM/nsf0OepZwLjt24FLgKO7VdizJSdpHlXzcTawvj7+TNL7gJ8Dr7a9WdJHgMOAu6gy7UsBbF8kaYwqsd0JvB54fP3aibbv6DjfMqCzCRsRrWYmJhqZXV1D1UX9FnAocPM0ZW4BnlI/XkiVk2bUT0tuX2AFcCxVK2w+sNr2UcBvgOMkvQSYbfsI4PPAO6eryPaHgdOAi2yPdSa4usy47YU9mrQR0SZurCV3OfBaSecBJwHrJJ3VUWYlcLSkq4E3A+d2q7CfMbkHgFOoWmB7UjUlr6/fWwv8ATBrymvXA8cDN02pYw4RUbYGNs20vbHu5S0Bzqm7pDd0lLkbOLHfOvtpyb2BqnX2SuCe+rVn138/E7gVWAccXr92eP38fmD3+rVjp9R3L7AbgCT1G2hEtFe14mHwS0gAbN9pe1Wd4AbWT0vuCuBC4I318/2A+ZK+AdwBfNH2g5KWSrqGh8fkdgVWSXpaR33fBd5Tl10BXNbA94iIEdtul3XZvho4pI9yp07z8pHTlLufelIiIgphM9HSZV1ZuxoRjdhuW3IREb1kF5KIKNvkzEMLJclFRAOyM3BEFM7tnHdIkouIBpimlnU1LkkuIgaWiYeIKF6SXEQUzP3uJzd0SXIRMTinJRcRpUuSi4hSGZhId3X718adoebM2b13oSH62YZGdsdpzB5z5446hB1D//d4GLokuYhoQFY8REThkuQiomhJchFRLBucTTMjomQtbcglyUVEEzLxEBGFS5KLiHJlWVdElMzkYuCIKJpxNs2MiGKluxoRpWtpjkuSi4hmZEwuIoqVezxERNlaPCa30zBOIulMSWPDOFdEjIKZmJjoeYxCWnIR0YjtdkxO0lXAd4Bn2j5G0m7AxcATgP+x/RZJ84BVwGxgve3XS9oD+BwwCxBwlaT3ATfZvlTSXwE3275sm3yziBiealBu1FFMq5/u6uHAdbaPqZ8vA260fSSwr6RnAvsCK4Bjgf0lza/Lfdn20cAD9WcvBl5ZP14KXN55MknLJK2WtHprv1REDNdkjut1jEI/Se5G2/885flTgZfXLbynAPtRJbFTgE8DewJzgN8H/rv+zGoA2/8H7F6Pz91o+77Ok9ket73Q9sKt+kYRMRK2ex79kLRS0rWSlvcoN1/Sd3vV10+S29Tx/GbgfNtjwHLgJ8AbgM9TtdLuqcutBw6qHx825fOXAf9I1aqLiBLYTGye6Hn0Iul4YJbtRcACSQd2KX4uVYOqq62ZXf04cKykq4E3Aj8FrgDeDVxZl9mvLndC3eJ77JTPf56qdfvNrTh3RLRUny25vSaHo+pjWUc1Y1Tj+1Dlk8XTnUvSC6gaVD1vD9dz4qFusU19fg9wUkexq4FDpvn40R2BHQz8E3C223pRTURssS24GHhDj6GoucBt9eONwAGdBSQ9Bngv8DKmGdfvNNRLSGyvA54zzHNGxHA01G7ZxMNd0HlM39t8F7DC9l393At5KBcDR0Tp+pha7S8JruHhLuqhwK3TlHkR8JZ6KOwwSZ/oVmEuBo6IwRnczIKGy4FrJC2guiTtZEln2X5oprW+fA2oruO1fUq3CpPkIqIRTSzbsr2xvsRsCXCO7duBG7qUH+tVZ5JcRAysyV1IbN/JwzOsA0uSi4jBtXgXkiS5iGiAt98F+hERfUlLLiJKZpLkIqJQtpmY2DzqMKaVJBcRjcjEQ0QULUkutol777171CE8wh5z5446hEdo4w9eP+stt0dt/LeGJLmIaEC1ldJoblTTS5JcRDQiSS4iipbuakQULUkuIgqWMbmIKJizQD8iSpckFxEFM25g08xtIUkuIhphkuQiomDprkZEsTLxEBGFc5JcRJQt+8lFRNHSkouIclWDcqOOYlpJchExMJN7PERE4bJ2NSIKltnVvklaBiwbdRwRsWUmsqyrP7bHgXEASe381RARj1DNOyTJRUSx2ttd3WnYJ5R0kKSzhn3eiNjGJi8j6XaMwNBbcra/Dywf9nkjYtvKJSQRUbS2dleT5CJiYLazdjUiypaWXEQULUkuIorWVJKTtBJ4OvAV24+6EkPS44DLqPLXJuAVtu+fqb6hX0ISESUyeKL30YOk44FZthcBCyQdOE2xVwPn2V4C3A4s7VZnWnIRMTAbJvpb8bCXpNVTno/Xq5wmjQGr6sdXAouBHz7yXL5wytO9gV92O2GSXEQ0os/u6gbbC7u8Pxe4rX68EThgpoKSngfsYftb3U6YJBcRDXBTa1c3AXPqx/OYYUhN0p7AR4ATelWYMbmIaITtnkcf1lB1UQEOBW7tLCDpMVRd2nfbXt+rwiS5iGhEQ0nucuC1ks4DTgLWTbPW/Q3As4EzJF0l6RXdKkx3NSIG1tR9V21vlDQGLAHOsX07cENHmY8CH+23ziS5iGiAsZtZ1mX7Th6eYR1YklwUTdKoQ3iUtq0MaOrfqG3fa1KSXEQ0IkkuIgrW3p2Bk+QiYmC5x0NEFC8tuYgomHFuSRgRJcs9HiKiaBmTi4hiNbXiYVtIkouIBuQSkogo3EQmHiKiZBmTi4hyVYNyo45iWklyETEw095LSLZo00xJ8yW9YGtPJumEelfPiChMQ5tmNq7vJCdpH+Bc4GWS/rfekfMqSYdJ2kfSFZKuk3ROXX5fSV+VdK2kv6mruQm4UNKuzX+ViBgle6LnMQp9JTlJC4BzgFOBXwPvtz1WH98D3gd8AVgELJa0CHg7sLK+f+JhkvaxvY4qUa5IoosoiZmYmOh5jELPJCdpP+Bs4K2275qh2HOBq121R68F/ojqtmKvkbSf7aX1NsbYvgn4IFWim93El4iI0Zq8GHh77a6OAWttb5zy2hlTuquzgN2Be+r3fgs8FrgA+Dfg65Le01HnLcDvgIM6TyZpmaTVHTegjYiW226TnO1PAxslnTrl5and1c1UN4GdV783t35+CLCS6rZix0g6CqBOiucDl9heO835xm0v7HED2ohoFYMneh8j0NeYnO2LgA2STp+hyPXAmKrN4p8PfBtYDjzP9r3AD4DZknYBVgCfsn3doMFHRHu4jz+j0Pd1crYvlXQ81ficJZ1Sv/VR4EzgEuDVwDW2r5N0NzAu6UHgR8AVwBnAx+rJiogoSFvXrqqtgQFIam9wEVupbT9zktYMOjw0a9bOnjfv8T3Lbdz4q4HPtaWy4iEiGtG25D0pSS4iGpEkFxFFS5KLiLIlyUVEqWwz4c2jDmNaSXIR0Yh0VyOiaElyEVGw3MgmIgqXezxERLFy39WIKJxb25Lbons8RETMpKntzyWtrG+bsHyQMpOS5CKiEU1smlnvdDSrvm3CAkkHbk2ZqdreXd0ArG+gnr3qutqkbTElnu4ai6fadrERTcX05Abq+KrtvfooN7tj1+9x2+NTno8Bq+rHVwKLgR921NFPmYe0OsnZ3ruJeiStbttOw22LKfF017Z4oF0x2V7aUFVzqe4PA9UO4wdsZZmHpLsaEW2yCZhTP57H9DmqnzIPSZKLiDZZQ9X9hOr+MLduZZmHtLq72qDx3kWGrm0xJZ7u2hYPtDOmQV0OXFPf6/lY4GRJZ9le3qXM4d0qbPX25xGx45G0B7CE6l7Ot29tmYfKJslFRMkyJhcRRUuSi4iiJclFRNGS5CKiaElyEVG0/wdYm1uJLJ9RFwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "108m 7s (- 66m 16s) (3100 62%) 0.0251\n",
      "111m 39s (- 62m 48s) (3200 64%) 0.0289\n",
      "115m 5s (- 59m 17s) (3300 66%) 0.0266\n",
      "118m 36s (- 55m 48s) (3400 68%) 0.0253\n",
      "122m 7s (- 52m 20s) (3500 70%) 0.0234\n",
      "> 我 找到 你 了 . 没有 么 ?\n",
      "= i found you , didn t i ?\n",
      "< i found you , didn t i ? <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAD8CAYAAAAMs9NCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAGHRJREFUeJzt3X20ZFV95vHv020DzYsthh46zQrGSMeAEphF+4a8XFC0WegoqIFEdPkWkhiJrBk1KhBxBtEQh+BEUBvadADFAWdCFHEcDHZs04B0M8CAoqIDOjgdadPatLwofZ/5Y58LZXlfqu4591bV6efDOotTVfvsfaq66nf32fvsvWWbiIi2WjDoE4iImEsJchHRaglyEdFqCXIR0WoJchHRaglyEdFqrQtyknaVpIbyWtL1eIGkxj8zSSsk7SZpd0nLGsz3SU19FvNF0tO7P/cZ0u/TcPmt+03s7FrxDyppoaQxSW8F3gu8UtJLZpnX1zse3iDpWZLeWz0+EviypLslfVnSP0p6T8ex62f5o/so8HTgeODs2Zz3FP4YWC9pXdf2oKQvN1hOk34b+FIV7C6SdGP1Wd8zRfrrJO3fRMFVgNsg6RlN5Nc0SftKOrbG8a+StEuT5zQKnjToE6hD0l7AZ4ClwE+AvwKeCjwCvFnS/rYv7TPbRzr2HwW+Czws6STb/x04VtItwAXAF21b0juBK4EdwPY+38PBwGbb35T0n4HlktZ1JHm97e/3+R4AsP1RSgCdKGtf4CPAfwX+fDZ5zoPrga3AocAewO/Z/kHnZyLpn4EHgV2BZwKrOyqsuwCX2L5yFmWfDnzf9nclLbL9i9m/jV8maTHlc18KfMP2m/s8fhnl+71V0kXAv1QvnQFsBi4H9gTW236XpF8H1gJ7AV+1/W7gbuBiSX9q+9EG3tZosD2yG6UmuggYA86pnnsP8GLKD+AFwIIe8zoCWEf5gf236rnrq7z+A3Bp9dzLqnR7AuspQXUtpSa2Dtitj/NfBNwCnFOVv5byI10OLAZuavCzeinwDeAlg/53m+Yc9wO+Buxbvf+vAape+zpwOLC8I/0ngD9oqOznAd8Gfg14OfCFht/bGzq+o3cAh/Rx7HLgMuAp1Xfl1K7XPwG8FRCwofqcPkT5AwHwP4Bl1f7vAJcCuw7633u+tokv0EiS9GLgHcASypfz28BvUmpjmyk11f9i+7o+8lwH/CFwCbAC+ALli3MLsBvwT8AVwC+Ao4BvVuWdQwlSq2w/Qg8kvQl4DXAzcACldvUk4Fzg7cDv2l7X67nPUNZpwFNtf6iJ/OaKpJcDJwAPUIL8F6rnX0L58X7W9p2STqT8uO/oOHwJsMH22/ss82Dgs8CrgO8Bt1KC5611309HGb9F+Z5spXxPD7P9ox6O2w/4APBntrdJOge4x/YVHWluowS+OyV9GPgBMA4cB/yJ7fu78lxB+a69rdfv6igb6ctV21+mtJGNUWpzf0kJGguBq4CP9BngllLahF4PHAO8jxKADqb8pb8K+HfAM4DnVNvvAP9+lm9hLeVHNWb71OqHvJVy2fsSSk2yKeMN5jVnbH9e0hcptbrPSHob5fPYnfIH605Jr6DUVO4FLuw4/CBKLbBnklYCnwJOrvJeA/yV7Vsl7eqGLutsf68q72Lgr3sJcJUx4Fbb2zqeO1PSW6r9F1EuSX9WPX4IeDLlD+VjwFckrbV9Xsfx91CaYg6iBPRWG+mOB0nLJb0eOA34A+Bi4FuUwPMOSoDqNa8TgL8F7rV9tksVdxNwKuUv4uWUS6gTgVWUAPdhShvLrNgep3wRkfQUSpCe+LJup7Sf7YyupvxYPwFca/tlwMOUjqDnUP69X0MJ3I90bD8H+r00uQ04xvZtkk4Hfmh7jaRnU2p3jZG0EDjQ9gW9HmP7U8C26twmfMD2WLXtALZRmk+gtGNuA54NrAEOAV4q6eiOc7gQuKLJmuowG+kgBywDfgO4Cfi07TdS/jK9HHi27Uv6yOu66sf0847nfgicDLyJ8uPZCnyQEvBuAt5PuYxdRKk91vFJ4N2ULyi2rwWWVJfkOw1Jvwtst/0ApVlglaSPApfb3mb7FtsnUD6nZ1A+s4nttf2WZ/sx2z+U9GbgabbPrp6/E3hQ0qnNvDOoAtKLZnHcWmCLpHdNkeRmYKy6XeiFlPbLs4AX2H6Ycnm8m6RFwEWUz/LGWbyFkTTql6u3ArdKOgY4qqoNnU25vLlb0oHAwbav6iGviRqAqr92b6AEt8uBV1P+Ml5GuWx6IbCS8hfxx7ZfWx24kP5rEgsoDcZ/bvs7VXvJhDdReo1rkfQnwO9TevfmhKSDKO1YZ9XIYwGlNv5H1VMHU/7oHAr8RNIy25s7Drm5+sM0cfzRlI6hfst9HyVI3ihpI6VtbxfKv81Rkj5v+6ezeU9d5SwFzqT0iPbF9pWSTgLOA9xxufoxSnvwFZQgv972jZIepPQ6P0ZpErm+Kvvjtm+r+15GyqB7PoZl44meqYspAe0DlKr/IuBzlHaMl1K+VH9EuWfu33QcfyXwv2dR7rHAedX+M4H7gD9s+L29gvIDGOoeNWAf4D9V+5dTLldXVI9PpPS2HlY9PoByKTtx7HOrf6NXzqLc51P+kD0H2B9Y3PHa2cBzB/3ZZJv9NtK9q/NJ0pNsPzbN68uAf3HND3SmciKiPwlyEdFqo97xEBExrVYFueqG15HNfz7KSP7tzn++yhglrQpylPunRjn/+Sgj+bc7//kqY85UExGsn+b1RZKulbShGjU0rbYFuYgYYZL2Bv6OcmfDVE4HNto+HHhZNVHH1HkOc8eDpDk9ucMOO6yv9A888ABLl/Y+wGHTpk39nlLEIGyxPeuRO6tWrfKWLVt6Srtp06a7+OWZflbbXj3xQNKTKbdz/YPtscnykPQ54N22vyHpHcAm21+ZqsyRvhm4ro0bN85p/hqt+Spj53VfnYO3bNnCLbfc0lPaBQsWPGJ75VSvuxqjO8NvZw9gYtKBbcwwXnmnDnIR0Yzx+b0i3E4ZR/5TypjdaedwTJtcRNRieh851ZBNlPkXoUxAcO90iVOTi4iajPsest2barr3g1xmuZ7wd5Rp74+kTBc17WxDCXIRUY9hx3izQW6i08H2DcANXa/dJ+k4Sm3uL1xmd5lSglxE1GLmvU0O2z+kTGI7owS5iKhtmG9FS5CLiNqGOcgNpHdV0oUzp4qIUWCb8R63QRhITc523zOjRsTwSk2uS9fiyRExwgzssHvaBmHo2uSqaWJGehaFiJ3NMNfkhi7IVYN1V8PcD9CPiGYMqr2tF0MX5CJixDQ7ZKtxCXIRUcvE2NVhNaje1bFBlBsRc2PH+PigT2FKqclFRE1zN0C/CQlyEVGLDQ2Pz29UglxE1JY2uYhotQS5iGitQUy11I8EuYiox07vakS0Wy5XI6K1DLmFJCLaLbeQRESr5XI1IlotQS4iWsvpXY2ItktNLiJaKzcDR0Tr5RaSiGi1Yb6FpKfVuiQtkXSDpHWSTmyiYEnnSBprIq+IGBzbjI+P97QNQq81uUOADbbPmsuTiYjRNNJtcpLeDrwReIqkI4CTgQuB5cD/rV57L7DO9jpJb6gO/U1gEXAEsARYBTwKXA0sBASsa+6tRMSgDHPv6oyXq7Y/ApwBrK3WZngNcKfto4FvA2+a5vADqnSfBo6lrKd6re1jgF9MdoCk0yRtlLSxr3cSEQPjasWumbZB6KlNrstBwM3V/s3AgV2vL+7Yv6z6/4+AXYCnA3dUz00axGyvtr3S9spZnFtEzDPbjPe4DcJsgtxdwPOr/edXj38O7FU9t6oj7c+6jr2PEiQBDp1F2RExhNzjf4Mwm1tILgXWSvoq8APgPOC3gYslnQD8eJpjLwGulvRqSntdRIw4AzuG+B6SnoKc7XVUnQS2HwV+vyvJncBR0xy/tuPhMf2cYEQMv5HueIiImEmTbXKS1kjaIGnSW9Yk7S3pOknrJX18pvwS5CKinh57Vnup7Uk6CVho+3BguaQVkyR7HXCF7SOBvSRN20mZIBcRtZi+biHZZ+IWsWo7rSu7MeCqav8Gyn223X4MPFPSU4DfAL4/3fll7GpE1NbH7SFbZrg9bA/g/mp/G3DAJGm+BpwA/BlwN7B1ugJTk4uI2hpsk9vOE/fa7snkMeo84I9t/0dKkHvjdBkmyEVELRPzyTUU5DbxxCXqIcC9k6TZHThY0kLgedUpTGmnvlyVNKf5z0e3+ly/h4gZNTtk6xpgvaTlwPHAKZLO7Zoc5IPA3wJPA24Erpwuw506yEVEM5oasmV7WzUF23HA+bY3A7d3pfk68Kxe80yQi4haJnpXG8vP3soTPay1JchFRG1ZrSsiWmxwg+97kSAXEbXYZRtWCXIRUdtIT38eETGTYZ6FJEEuImrJ4tIR0W7VkoTDKkEuIupLTS4i2syjPv15RMR0hrgi11yQk/R+4G7bV0p6H2X2gFXMsAh11/oPVJPodU+kFxFDqtwnN7xRrsmpli7jiQVuVlGWKOx1EerHZd3ViNHTtsWlJ2X7u5T51scoq3f1swh1RIwsM75jvKdtEJqeNPMzwCcptbp+FqGOiBE1cbna+ppc5bOUewO/RlmE+lnVItQrgLXA54B3VsuITbcIdUSMkGEOck12PDyLMlvneS7vpu9FqCNiRA1xx0NjQc72XcBzm8ovIkbHEMe43CcXETXZA+tU6EWCXETU0vT0501LkIuI2hLkIqLVEuQior1syAD9nVMWfh68LPA9P1KTi4jWMjCemlxEtNaQz0KSIBcRtWXSzIhoscGNS+1FglxE1JYgFxGtNewzAyfIRURt3pEgFxEtlppcRLTXACfE7EWCXETUliAXEa2VqZb6lHVXI0aMwUM8aWbTC9nUlnVXI0ZNb4vY9Frbk7RG0gZJZ82Q7mJJL58pv6ELchExesq9cjNvM5F0ErDQ9uHAckkrpkh3JLDM9udnyjNBLiJq66Mmt4+kjR1bd9PUGHBVtX8DcER3WZIWAZcA90p6xUznNu9tcpL2A95p+4z5Ljsimmf3NUB/ywxNUXsA91f724ADJknzeuAbwPnA6ZL2t/03U2U4iJrcZuD/DKDciJgjDbbJbQcWV/t7MnmM+rfAatubgSuAY6bLcFCXq58YULkR0TgzPj7e09aDTTxxiXoIcO8kae4BfqvaXwncN12G8365ansHsGO+y42IOdLsAP1rgPWSlgPHA6dIOtd2Z0/rGuCTkk4BFgGvni7DobtPLiJGUEOTZtreJmkMOA44v7okvb0rzYPAa3rNM0EuImopIx4azM/eyhM9rLUlyEVEbRnWFRHtZTM+xMO6EuQiorbU5CKitTILSUS0W9M9Dw1LkIuImjIzcES0nIe33yFBLiJqMr0O2RqIBLmIqCUdDxHReglyEdFi7mc+uXmXIBcR9TQ7C0njas8nJ+mcatYAJF04U5qIaKGmFnmYA43W5DKlecTOx8B42y5XJe0NXA0sBASsq55fZ3tsqjSSzqFMcncEsARYVc0XFRGjqr81HubdbC9XTwOutX0M8Is+0xxg+2jg08Cx3QdJOm1iJZ9ZnltEzKtm111t2myD3NOBO6r9qYLRVGkuq/7/I2CX7oOyuHTE6GljkLsPOKjaP7TPND+bZZkRMaSGOcjNtuPhEuBqSa+mtLHNNk1EjDgb3LZJM21vYZK1Dic6HaZJs67j9bWzKTsihs8Q3yaXm4Ejoq5MtRQRLZcgFxHtNeTDuhLkIqIWM9w3AyfIRURNxpk0MyJaK5erEVP76UMPzWn+ixfvNaf5RzHEMS5BLiLqS5tcRLRW1niIiHZLm1xEtJuzJGFEtFva5CKivUqj3KDPYkoJchFRy5DHuPqrdUVENDlppqQ1kjZIOmuGdPtK+l8z5ZcgFxH12IzvGO9pm4mkk4CFtg8HlktaMU3yDwOLZ8pz3oOcpEMlTTVlekSMoD5qcvtMLFRVbad1ZTUGXFXt30BZ2e9XSDqWspTCjKv9DaJNbiLA3TaAsiOiYX3eDLxlhkWq9gDur/a3AQd0J5C0C/AXwCuBa2YqcF6DnKQPAidW+6+z/aL5LD8i5kaDNwNv54lL0D2Z/Grz3cBFtn8iacYM5zXI2X6PpG9V+2snS1NVX7ursBExtNxk9+omyiXqTcAhwLcmSfNi4FhJfwocKulS22+ZKsOhu4XE9mpgNYCkIe6YjgigxLjmBjxcA6yXtBw4HjhF0rm2H+9ptX3UxL6kddMFOBhMkHsY+DUASfIwD3qLiJ40NazL9jZJY8BxwPm2NwO3T5N+bKY8B3ELyfXASZL+GThyAOVHRIMmOh6auk/O9lbbV1UBrrZ5r8nZ/lfKNXVEtEFmIYmIdnMG6EdEy6UmFxFtZhLkIqKlbDM+vmPQpzGlBLmIqC0dDxHRaglys3TYYYexcePGOcu/l3FvMbeW7L77oE8hGpAgFxGtVW70zUI2EdFiCXIR0Wq5XI2IVkuQi4gWS5tcRLSYM0A/ItouQS4iWsy4oUkz50KCXETUZoY3yA1kcWlJFw6i3IiYG03ODNy0gdTkbJ8xiHIjonnD3vEwqJrcukGUGxFzobda3E5Vk5tO57qr+++//4DPJiJ6MczzyQ2kJjcd26ttr7S9cunSpYM+nYjoQWpyEdFepVFu0GcxpQS5iKjFZI2HX9HLqtcRMToydjUiWmxw7W29SJCLiNrGM6wrItqq9DskyEVEa+VyNSLaLkEuItost5BERKvlcnWWNm3alAWgY+jN9Q982H8Dtod67OpQB7mIGA2pyUVEqyXIRUSrNRnkJK0BDgSus33uJK8vAT5DiV/bgZNt/3yq/IZuqqWIGDUGj/e2zUDSScBC24cDyyWtmCTZa4ELbB8HbAZWTZdnanIRUYsN472PeNhH0saOx6ttr+54PAZcVe3fABwBfOeXy/PFHQ+XAj+arsAEuYiorY/L1S22V07z+h7A/dX+NuCAqRJKegGwt+2bpiswQS4ianKTY1e3A4ur/T2ZoklN0lOBvwFeNVOGaZOLiNoanP58E+USFeAQ4N7uBJJ2oVzSvsf2fTNlOO9BTtJiSZ+TdGPVixIRI67BIHcN8DpJFwC/B9wlqbuH9c3AYcCZktZJOnm6DAdxuXoycKvtcyTdIekQ27cP4DwiogFNrrtqe5ukMeA44Hzbm4Hbu9J8DPhYr3kOIsh9FfifkhYDewP/bwDnEBGNMXZzw7psb+WJHtba5j3I2f4egKSLgb+2/Uvdv53rrkbEaMiIhy6SFgIH2n5r92vVPTOrq3TD+8lFxOMS5LrY3iHpRYMoOyKaNtwzAw/kFhJJS4ELBlF2RDRrYo2HXrZBGFRN7gHgjEGUHRHNG+aaXEY8RERNxlmSMCLaLGs8RESrZd3ViGitJkc8zIUEuYioabhvIUmQi4jaxtPxEBFtlja5iBab63VRh35d19Io18zJzIEEuYioxeQWkohouXQ8RESrpU0uIlrM6V2NiPbKzcAR0XoJchHRYoa0yUVEmw3zLSR9zQwsaV9Jx862MEmvqhaGjYgWaXDd1cb1HOQkLQM+DLxS0jerRV3XSTpU0jJJ11cLRp9fpf91SV+StEHSh6ps7gYulrRr828lIgbBNuPjO3raBqGnICdpOXA+cDrwr8AHbI9V223A+4G/Bw4HjpB0OPB2YI3tw4FDJS2zfRclUF6UQBfRHiNdk5O0H3Ae8DbbP5ki2fOAr7q8iw3Ac4D7gVMl7Wd7VbUSNrbvBv6SEuh2m6S80yRtlLRxdm8pIubbSAc5YAy41fa2jufO7LhcXQjsBfyseu0h4MnAR4EvAl+R9N6uPO8BHgUO6i7M9mrbK22v7O+tRMSgjHSQs/0pYJuk0zue7rxc3QFsA/asXtujevxsYA1wCPBSSUfD4wtLXwhcYfvW5t5KRAzMxEwkM20D0FObnO21wBZJ75oiyc3AmMqcLS8Evg6cBbzA9sPAt4HdJC0CLgIut31j3ZOPiMGzzbh39LQNQs/3ydm+UtJJlPY5S3pL9dLHgHOAK4DXAutt3yjpQWC1pMeA7wHXA2cCH686KyKiJYZ5xIOG+uSk4T25iHkyD5NmbqrTBr5gwULvssuv9CFO6tFHH6pV1mxkxENE1JSFbCKi5TKfXES0VqZaioiW81DX5PoaoB8RMRl7vKetF5LWVGPez6qTZkKCXETU1tSIh+o2tYXVmPflklbMJk2nYb9c3QLc10f6fapj5spc5z8fZST/Ect/Fuui9lvG0/otoMuXbO/TY9rdusalr7a9uuPxGHBVtX8DcATwna48eknzuKEOcraX9pNe0sa5vAdnrvOfjzKSf7vzn68yOtle1WB2e1Am94AyPPSAWaZ5XC5XI2KYbAcWV/t7MnmM6iXN4xLkImKYbKJcfkKZ3OPeWaZ53FBfrs7C6pmTDHX+81FG8m93/vNVxly5BlhfTdR7PHCKpHNtnzVNmudPl+FQj12NiJ2PpL2B4ygT8W6ebZrH0ybIRUSbpU0uIlotQS4iWi1BLiJaLUEuIlotQS4iWu3/A04P116j033EAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "125m 38s (- 48m 51s) (3600 72%) 0.0241\n",
      "129m 8s (- 45m 22s) (3700 74%) 0.0257\n",
      "132m 35s (- 41m 52s) (3800 76%) 0.0298\n",
      "136m 4s (- 38m 22s) (3900 78%) 0.0241\n",
      "139m 33s (- 34m 53s) (4000 80%) 0.0239\n",
      "> 我 是 为 你 来 的 。\n",
      "= it s you i ve come for .\n",
      "< it s you i ve come for . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASEAAAD8CAYAAAA4yhJeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFyhJREFUeJzt3XuUXVWB5/HvjzQIhqcGCWFse9lkFLANLbHFGKFEkDi23TYMDx/Yajus7lEea9pxEBin7eYlw7hwFNRqonGM4kTbZtRxbHFCFkgimKShJYjgIziNHTUKxqCC5P7mj3NqpVJU1b1VdW7tW/f+PqyzuI9Te5+bpH53733O2Vu2iYgoZa/SBxARgy0hFBFFJYQioqiEUEQUlRCKiKISQhFR1G+VPoAmSXoK8Lgbvu5A0reAh8a8fJTtI5qsJ8Yn6XLgBtvfl7QPcKPt00sfVzRjzoeQpHnAS4GjgcOAuyQ9avsrDVbza+CrY15rJIAkCXie7W9Keiew1faaJsoeVcdi4P9RtXwPtL2tyfJH1XMB8KjtGxou+jjgP9ePTwF+Kem59fPv2X684fqmRdJhwDG2107z508HvtArn2e2zNkQknQA8GngUOAR4L8CT6MKjD+T9NsN/zKMDaHXNVTufGBY0mXAAuCuhsod7YPAhVRBfRLwtqYKlvQR4LmAgUVAS9IbAAH/ZPu8GZS9F/BT4G5gs6QvAscCO4CLgD8ATgPum9GHaICkhVT/Bh+WdB3wo/qtC4FtwCeA/YHbbL9T0uHAKuAA4FbbF1F9juslvc32Y7P9GUqZsyEEPAq8BngJMGT7ZklLgd8AbwReIGkv260G6voecM2Y13403o7T8DjwJ1Th8HTghw2VC4Ck3wO22f6WpP8GLJK0btQub7T9gxlUcT7Vn/ca4LVUXwK3AK8GhmdQLrZbkjbbfrmkIeDfA/8C/KXtHfUve/FWg6RFwFXAeVShc7nt1aPe/wjw98CHgNslLQP+CFhpe42kL0taaHuLpGuA6wYpiOZyCJ0EvAM4CHh6HUC/A5xO9c3zW8B/B7403QokHQWcC3y/3sa+/z7g87bXTbcO4HpgMVVLYinwu1UPDai+9Z9je1qBJ2lv4KPA/5a0HPgxVXAvAB4GbplhAGH7sXrM7D9QBQRUrZSPNNStWFKH5sHATVQtitVUv8RPoQq9YiQdAVwOvL0OxvF2exHwAduWtB54IdUY4xsk3W57xciOtu+T9F6qIHq77aKfbzbM2RCy/VXgq/U35BDwXuAMYB7Vt/L7bU87gGrfBf4G+DfAPcDhVONPV9T17APsnEkFtt8KIOkg4MvAKbZ/U7+2mSo4puscYHv9+M+B/0T1GS4DLqAKi2mTdBLwHuAxYG/gSKowfQC4pg7BK21/eQbV3GX75JG/Z9vflfSEpOcD+1E4hKj+7W22vWPUa5dIemv9+OVUXa5H6+e/BA6k+jt4ArhF0irbV4z6+e9Q/ZkeDWzu4rH3hDl7il7SIklvpGqpvI6qRfFtqvGJdwB3zLQO24/b/hnVt/wu4FfAmcDfAZ8BXmb7VzOtp/bvgBuBj0t6d30WSDM807eKKpyx/QbgGOAZVJ/lFcA3Z3LAttfafqntk6m6rLcB3wI22T7Z9okzDKCJvB3YAjwV+MV0C1HleZqg+dIJ258EdkgaPfZ1ue2hettFNYa1f/3e/Pr584CVwBLgVEkn1sc0D7gWWG277wMI5nAIAQuBZwJfBz5l+81U3xqvpjrb9LdNVCLpVVThs5LqF3e17VNtv6Kps1j1uM0ZVGMGfwrsCxwF/Hwm5dbjYU/UdRxMFUgj38g7gffPpPy63HmSPkDVXV0HfBLYR9JqSQtmWj7w+3V37NqRF2z/sP7lXjDSapym84Hn1P+fNturgO312c3x3AEM1WH3EuBO4FLgxfWX2P3AvnXL8TrgE7Y3zOSY5pI5G0K2N9u+nOrb3PUv2VVU3/L3STpK0pkzqUPSs6gGpM+iGjzeG1gh6U5J35T0L5IunWEdLwM+BZxt+zf1djFVV2NG4zVjfJSq+7UDwPYXgYMknTzdAiX9a6ovgUds/zXVGTFsn091pueBkW/4aZY/j6pVNQT8BVVXBkl/KukBYOt0y67tTTU+NqOwB7B9I1U36myq7ti6ejsL+Cvgj4ENVGfHNlB186+UdGt9HDdT/f182PbGmR7PXKLMJ1ROfVblU8CbbG+tX9uL6ptzX+C8GQ56I+kEqnGJ1bYfqK8ZusT2myQdShUg02pN1BeHLrf9f+vnfwn8wvZw/fzZwPebvni0SZKeYXsm424xQwmhiChqznbHIqI/JIQioqi+CiFJ56b8snWk/LLlz0V9FUJU1wyl/LJ1pPyy5XedpMMk3TbJ+3tL+qKk9ZLe0q68fguhiOgiSYcAH6e66HIi5wEbbS8D/rC+2XziMnv57Jik3j24iNpxxx3X8b4/+clPOPTQQzvef+vWrWzfvn3aV3QDrFixwtu3b2+/I7Bp06Yt7HkrzPDIJRcAkg6kuh7sf9XXbz2JpM8DF9m+V9I7qK71umWiOufsvWMRvWLjxu5dW7h06dIZl7F9+3a+8Y1vdLTvXnvt9WvbE1Y6co9cmztd5rN7EsAdVPN8TSghFDEAWrPb49lJdcX/z6numZv0Ju+MCUX0OQO2O9oasglYXj9eQpvba9ISiuh7xnSnJVRP53K07Q+OevnjwJckjUy7POmMFgmhiH5n2NVqNoRGBqXr+bTXjnnvQUmnULWG3l3PeDChhFBEnzOzPiaE7R9STS7YVkIoYgD08qU4CaGIAdDLIdT1s2OSrh3z/FhJx3a73oio2KbV4VZC11tCti8c89JIAHVjfa2IGEcvt4S6HkKS1o2MpEu6kmqaVCSdY/vl3a4/YtAZ2DXIITSa7XdJ+nb9eNV4+9RTHcz5O40jeslAt4Smqr5ZbmSO4t79k4uYQ0qN93SiRAj9imq5YyTNdF2tiGin2VsyGlfi3rGbgdMk3U61mmlEdFGBe8emZDbOjg2Nef4zYNprXUXE1O1qtUofwoR6bkwoIprWvRtYm5AQiuhzNjR8/2qjEkIRA6CXB6YTQhEDICEUEcWUmMpjKhJCEf3OztmxiCgr3bGIKMaQU/QRUVZO0UdEUemORURRCaGIKMY5OxYRpaUlFBHF5GLFiCgup+gjoqicoo+IYmzTysB0RJSUMaFRJO0HfAY4ENgOnGn7idk+johBkrNjezoaaNk+QdIKYH/gkZE3s+5YRPMSQnvaDNwj6SvAA8Da0W9m3bGIZo2sRd+rSiz5swS43fYrgEPIsj8RXecO/yuhRAhtBc6XtB5YCGwscAwRA8PArpY72kqY9e6Y7UeAU2e73ohB1stjQiVaQhExy1r1uFC7rR1JKyWtl3TpBO8fIulLkm6T9OFOji0hFNHvOlwCul1rSdJpwDzby4BFkhaPs9s5wGrbLwUOkLS03eElhCL63BTXol8gaeOobfTlMkPAmvrxWmD5ONX9FHiOpIOBZwI/aHd8uWI6YgBM4RT9dtsTtV7mAw/Vj3cAR46zz9eAVwHnA/cBD7erMC2hiAHQ0JjQTmC/+vH+jJ8fVwB/bvuvqULoze0KTQhF9LmR+YQaCKFN7O6CLaG63GaspwK/J2ke8KK6+kmlOxaT+t1nH9vV8j/5lc90tfzjjxxv7LRZkrpex4x0MOjcoZuA2yQtAl4JnC3pMtujz5RdCXwMeBawAbixXaEJoYgB0MRtG7Z3SBoCTgGutr0NuHvMPncCx0yl3IRQRJ8bOTvWSFn2w+w+Q9aIhFDEAMhqGxFRULmbUzuREIroc3a19aqEUMQA6OX5hBJCEQOgl++iTwhF9LksfhgRZWXJn4goLi2hiCjJPbwEa0IoYgD0cEOomRCS9B7gPts3SvovVHfXrgAWAf9MdTv/xcA62+skvQnA9qpxysq6YxENqq4T6t0Uamoqj/8BvLZ+vAI4ALjH9onA/cBbOi3I9rDtpZNMrBQRU9TE9K7d0kgI2f4u1XyyQ8A9VKus3lG/fQdw1Jgf2Y+ImCWmtavV0VZCk5OafRr4KFWraAtwfP368fXzx6laSFC1liJiFox0x/q6JVT7LNV1UV8DbgCOkXQrsBhYBXwe+I/1MiA/bbDeiGijl0OoqYHpY6hmU7vC1Sd5jN1jRCPuAU5oor6ImKIeHphuJIRsbwH+oImyIqJ5PZxBuU4oou/ZxQadO5EQiuhzTU7v2g0JoYgBkBCKiKISQhFRjg25gTXmqu989x+7Wn7PLxzYJ9ISiohiDLTSEoqIYnr8LvqEUMQAyKRmEVFQufvCOpEQihgACaGIKKbXZ1ZMCEUMAO9KCEVEQWkJRUQ5BScs60STMyt2TNK1JeqNGFR9P7PiVNm+sES9EYOo16fyKNUSWjfJe+dK2ihp4yweUkT/MnhXq6OthCIhNJmsOxbRtM66Yp20liStlLRe0qVt9rte0qs7ObqeC6GIaF51rVD7bTKSTgPm2V4GLJK0eIL9XgostP2FTo4tIRQxAKbQElowMhxSb6OXZB8C1tSP1wLLx9YjaW/gb4Gtkv64k2PLKfqIPmdP6QbW7ZMMhcwHHqof7wCOHGefNwL3AlcD50n6bdsfmKzCIi0h20Ml6o0YVA2NCe1k9xLu+zN+fvw+MGx7G7AaeFm7QtMdi+h7ptVqdbS1sYndXbAlwNZx9vkO8Oz68VLgwXaFpjsW0e+au4H1JuA2SYuAVwJnS7rM9ugzZSuBj0o6G9gb+LftCk0IRQyCBiY1s71D0hBwCnB13eW6e8w+vwDOmEq5CaGIPlddMd1QWfbD7D5D1oiEUMQA6OXbNhJCEf0ua9FHRGlpCUVEMb1+F31CKKLfNTky3QUJoYi+19szKyaEIgaAe3dcOiEU0fdMJ7dkFJMQiuhzGZiOiOISQhFRkKcyn9CsSwhF9LssAx0RxSWEIqIUA61B7I5Juhi41/ZNki4Cfgj8EfAM4Ju239atuiNilKnNMT3rujm962epZl8DOBF4JnCP7ROAwyU9f7wfyuKHEU1rbt2xbuhaS8j2/ZKOkHQg8HPgXwHL6pnZDgaOAP5pnJ8bBoYBJPVufEfMIYM8MH0ncCHweapu2J22PybpD4EfdLnuiKgNcgh9Fvga8CygBXxM0pup1ix6XZfrjgjqdccGdVIz2/cCTxv10pndrC8ixtfDDaGcoo/of5nKIyIKSwhFRDm5bSMiSjK9fbFiQiii7xlnUrOIKCbdsYgorYczKCEUMQgyJhQRxWSO6YgoK2NCEVGWs+RPRJSVMaGIKCdr0UdEST2eQV2d3jUiekRT07tKWilpvaRL2+x3mKR/7OTYEkIR/c6mtavV0TYZSacB82wvAxZJWjzJ7tcA+3VyeAmhiAEwhZbQgpGFJurt3FHFDAFr6sdrgeXj1SXpJOBRYFsnx5YxoYg+N8WLFbfbXjrBe/OBh+rHO4Ajx+4gaR/g3cBrgJs6qXBaISRpX2AV1QoajwCvBz4MLAL+GXgzsAH4MfA4sBC4sd5WAQcBX7B95XTqj4ipaehixZ3s7mLtz/g9qYuA62w/IqmjQqfbHTsXuNv2cuDvgAuo1hQ7EbgfeAvwVOAM4PlUk9ovAd4F/M+6T/kaSU8fW3DWHYtomuvZ7jvYJreJ3V2wJcDWcfY5GXibpHXAsZJuaFfodEPouVTL+UDVsjkcuKN+fgdwFPAj2zuBB4FdgIDnAH9RH+B8qpbTHmwP2146SZMwIqbC4FZnWxs3AedIeh/VohVbJF22R1X2CbaHbA8Bd9l+a7tCpxtC9wEvrB9fXD8/vn5+PLBlgp/7NnBRfYBXAT+bZv0RMQWtVqujbTK2d1ANTn8deJntu21PeKq+/j1va7ohNAy8oG7RvICqNXSMpFuBxfXz8VwFvEPS7cAK4EfTrD8iOjQyMN3EdUK2H7a9xnZHZ746Ma2Badu/5slriL12zPOhet+h+vmb6v+/ajp1RsQ05S76iCjLuYE1IgpLSygiSjIJoYgoxDat1q7ShzGhhFDEAMjAdEQUlRCKrun2P65O7/+J3pYQiohiqgsRM9F9RBSUEIqIotIdi4iiEkIRUVDGhCKiIOcG1ogoLSEUEQUZZy36iCjJ9G4INbbumKSDJK2VtE7SnzRVbkTMXFMzK3ZDky2hJcD6yeacjYjZNxAD05IuoFpr7GBJy4GzgGsZtQ6Z7cfrOam/ATzf9qlN1B0R7ZRr5XSike6Y7fcDFwKr6jmlz+DJ65BBtRLHhskCKOuORTSv1drV0VZCt9aiP5onr0MGVTB9brIfzLpjEc3r5TGhboXQFsZfh2xnl+qLiIl0uvpqn4XQDXS2DllEdJmp5pju5L8SGjs7ZnsdsK5+/BhPXoes4xUZI6JZuXcsIgrq7bNjCaGIAdBunfmSEkIRfa4ac04IRUQx6Y5FRGkJoYgoKctAR0RR6Y5FRDFZiz4iiktLKCKKSghFRFFNhZCklVSzYnzJ9mXjvH8Q8GmqbNkJnGX78cnK7NYNrBHRMwxudbZNQtJpwDzby4BFkhaPs9vrgffZPgXYBqxod3RpCUX0ORtanV8xvWDMhILDtofrx0PAmvrxWmA58MCedfn6UU8PBX7crsKEUMQAmEJ3bPskEwrOBx6qH+8AjpyoEEkvBg6x/fV2FSaEIvpeY8tA7wT2qx/vzwTDOZKeBnwAOL2TQjMmFDEAGpredRNVFwyq1XW2jt1B0j5UXbZ32X6wk2NLCEUMgIZC6CbgHEnvA84Etkgae4bsz4DjgEvqNQjPaldoumMRfa6pdcds75A0BJwCXG17G3D3mH0+BHxoKuUmhCL6nrGbuW3D9sPsPkPWiJ4LIUnnAueWPo6IfpIrpqegviZhGEBS7/7JRcwhCaGIKKi3Z1ac9bNjko4eZ0Q9IrpkZI7pTrYSZr0lZPte4NLZrjdikPVySyjdsYi+Z5wlfyKipMwxHRFFZd2xiCimqSumuyUhFNH3evsUfUIoYgBkLfqIKCpjQtE1kkofwsDrZldn6dKJJjmcgmpQaObldElCKKLPmZyij4jCMjAdEUVlTCgiCnLOjkVEOblYMSKKSwhFREFuu8RzSQmhiAHQy6fopzSzoqTDJJ003coknV4vjhYRs6ihdce6ouMQkrQQuAZ4jaRv1QubrZN0rKSFkm6WtEHS1fX+h0v6B0nrJV1VF3MfcL2kpzT/USJiPLZptXZ1tJXQUQhJWgRcDZwH/Ay43PZQvd0FvAf4e2AZsFzSMuACYKXtZcCxkhba3kIVZNcliCJmz5xuCUk6ArgCeLvtRybY7UXAra4+xXrghcBDwBskHWF7Rb1aI7bvA95LFUT7jlPfuZI2Sto4vY8UEWPN6RAChoDNtneMeu2SUd2xecABwKP1e78EDgQ+CPwf4BZJF48p8zvAY8DRYyuzPWx7qe0G7tyLCJjjIWT7k8AOSeeNenl0d2wXsAPYv35vfv38ecBKYAlwqqQTAerQuhZYbXtzcx8lIiY0cid9u62AjsaEbK8Ctkt65wS73AEMqZpX4iXAnVTL+rzY9q+A+4F9Je0NXAd8wvaGmR58RLRnm5Z3dbSV0PF1QrZvlHQa1fiQJb21futDwF8Bq4HXA7fZ3iDpF8CwpCeA7wE3A5cAH64HsyNilvTNFdO2Pwd8boK3Tx6z7z1UZ8tG+5up1BcRzeibEIqIuSgT3UdEYZlPKCKKyVQeEVGYe7olNKUbWCNibrJbHW3tSFpZ3w966Uz2GS0hFDEAmrhiur5EZ159P+giSYuns89Yvd4d2w48OIX9F9Q/0y1zvfzZqGPgyp/i2m9TLf9ZUyl8Av9ge0GH++475r7NYdvD9eMhYE39eC2wHHhgzM93ss8eejqEbB86lf0lbezmPWdzvfzZqCPlly1/PLZXNFTUfKob06G6NevIae6zh3THIqJTO4H96sf7M35+dLLPHhJCEdGpTVTdK6huTN86zX320NPdsWkYbr/LQJc/G3Wk/LLld9NNwG31JIevBM6WdJntSyfZ5/h2haqXL2KKiN4i6RDgFKpJDLdNd5899k8IRURJGROKiKISQhFRVEIoIopKCEVEUQmhiCjq/wN1q//NCeNexwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "143m 0s (- 31m 23s) (4100 82%) 0.0239\n",
      "146m 28s (- 27m 53s) (4200 84%) 0.0221\n",
      "149m 56s (- 24m 24s) (4300 86%) 0.0223\n",
      "153m 27s (- 20m 55s) (4400 88%) 0.0210\n",
      "156m 52s (- 17m 25s) (4500 90%) 0.0218\n",
      "> 汤姆 现在 很 生气 。\n",
      "= tom is very angry now .\n",
      "< tom is very angry now . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARoAAAD8CAYAAACo2WuRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAGAxJREFUeJzt3Xu4ZFV95vHv2801NDdtbk1ARmESQC4JbWhakCOINJNgDEZA1CCI/fiMYnwyE0VhIkYuyjCMCSLkGC4qEUJ8oIOEIDjQNAoC3SATmiCDTqMxg9AKtK2i0OedP9Y+UBzOpc45e1fVqfN++tlP76patdauOlW/WmvtvdaSbSIimjSn2wcQEf0vgSYiGpdAExGNS6CJiMYl0ERE4xJoIqJxMzbQSNpB0pbV/kbjpPvPnTuqMY/hMEmbNVzGK4ffj26TtEm3jyF6y4wNNMBewPWStgWuk7RC0nJJT0q6D0DSpsCSan+JpO9WaZZLeqp6fFIkbTIc2CTNlfQxSaskfaPanhiRflPgEuD5ab7e8Y5pU+Aq4ChJN7e8F/dLOrfGcraX9EfV/p9Let8YSc+W9CcjnquZGoCqH7XDpvH8t83U116XMWsCvc72bVUtYY7towEkvRV4L3BCFYAOB74raTdgS+BTwPbA/wS+YftXUyj6IMoX6TXAA8A3gP9ie3l1DDdU/38bWFeVOw+4QdJwHlsAZ9r+X1Mo/yUk/QfgCmDQ9jWSTgTeYvtZSQcDR0y3jBaHA/sC1wHPVdvI43kVcBywj6QTWh8CVgN/VuPxNE7SjsB/B56SdBHw4+qhDwOPA1+m/H3vsP0RSTtR/h5bAitsnwY8DHxe0gem+Jmb8WZkoJG0C3AacJrtn1X3vY8SBI6x/ZykvYH3A1+nfBguB56g1Cx2mGrZtm+vAtr5tt8j6b9W5f898NGWdIuq+/8BOLEq9wDb/zDVssfwPHAu8P8k/TfAwM2ShoCtgX+ssawTgAWSDgJ+E3he0nuAjSnv8WXAF4GLKIGl1Ua2z6rxWBonaQHwaeBUSmA52/aVLY//DSXoXgx8S9Ji4C3ApVXQv0nSjrZXSzofuGi2BpsZGWgovyRPUmomp0i6A1gE3APcVtUczgWeBv4a+H3g48D66r73A/tJOtf2xyZTsKRFVbm7SboMeKh6aAvgRyPSfhT4ge1Hque9Hqg10Nj+oaS1wJ3Ah4ADgTe31GiOrKMcSQdQgsUB1e0PA0/bvqIlzXaUZuI/U37lN7RkMelmajdJ2hk4G/ig7XUttdFWBwIX2rakO4HXUT4D75L0LdtLhhPafljSZyjB5oO2n+3Ay+gZMzLQ2H4OOFPScB/TUuB84DxKU+EJYHfgGUoAugWYS/n12bP6tbkZOGMKZX9b0tGUGs3Jkk6rHvpFVZMa7r85DPhTYKmkFZRf/Z0k7QU8aruWTmpJ21BqLU/bvqMKbrdVtb5HKX03dZgHfELSXNutAQRJG9l+3vaTkl4NXEOpWb2QBNiG8sWcKQaA+2yva7nvdEmnVPuHU5pHP69u/wLYCjiLUsu8TdIVts9pef6jwK8o/Yv3NXjsvcf2jNyA36FUa7ei9JP8LuVLtTfwSeBQ4AJgJbA/pRayE/AJygdk2TTKng9cUe3/R2AXYBWl2vy7wH+qyv12y3MWAZ9t4H04DjgWuKG6fQOwOXB1dSy3AZvUVNZc4C5gOeVL83C1f9MEz9sUuL3m1y3gtYAa/Iy9Bzi12j8TeNeIx+8H9qn2/wflh2UfYJPqb3A7cGjLe/dXwEFNHW8vbzP5rNOplA/9Mkpg2RV41vZqYDtK8+pvgd8A3kHpBP4KcAhwPbBI0kmTLbQ6w7MvcKCkb9h+BDiqOpbFtu+jfACPpjTVGmX7721fM/JuYA2l+Xic7V/XVNYG2wfZHgA+B3za9oBbmghjeA7YRtJm412KMEkfAn6r+r8RLs3CtZI+MkaSu4EBlXbV6ylN9zMoweSXwCPAZpI2pvRbfdn2XU0dby+bkU2n6izSYsoZpn8C3kf5ZT9K0tbAfsBmwIWUKu4bbX+0eu4c4GBKIPjyFIp/P3AYpc/n65JeRwk0FwBHSzqc0ul7nqQ5kjZ2aeq1Hv9cyi9xnae857b8fzzwNUqN5hBJ19kemk7m1Zdpjkc0m1oe3wjY4Ornu5XtIUk/pbx3m1FqotO1MaVm+UwNeY3J9lWSjgHOAdzSdLqYUsu5Engn5azTXZJ+BgxKeh74PqXZfjpwie3vNHmsvUyjfC56XvXlfq3tyyXtSQk0H3fpAP0M5cN3F/Bj2w+NeO6elGrun9l+eJrH8UFKDekU4BWUqjLAybZvlbQSuJkS+Frf6I2Ai2x/bTrljziWFbbfIGlZdRyXUKrvF1J+SW+aZv4LKe/by05pVzYFTrT9/TGe/xbgC8CBttdM51ha8tze9hMTp4xum5GBpldI2tSz8FRlxGQl0ERE42ZyZ3BEzBB9HWgkLU3ZKbvfy54J+jrQUC7kS9kpu9/Lrl01kPSOcR7fWNINku6UdPJE+fV7oImISaoGJH+RMqxmLKcCK20vBv5AE0xR0tOdwfPnz/duu+025ec/+eSTbLfddlN67qpVq6ZcbsQUrLU9tQ8rsGTJEq9du7attKtWrVoNtI61GrQ9OHxD0laUK6//sbo482UkXU8Z1PxQNbB4le3bxiqzpy/Y22233Vi5cmVXyh5jEF1EUx6bzpPXrl3Lvffe21baOXPmPGt74ViPuxrfNcF3oHUQ8TommBGhpwNNRLRvqLOtk/WUC0KfoQy4HXe4TfpoIvqAaX+AdE1WUYbyQLnyfc14iVOjiegLxjRTo6mmPNnL9uda7v4icKOkQyjTXtw9Xh4JNBH9wLBhqN5AM9wRbPtW4NYRjz0m6QhKreYvxhpsOyyBJqIPmI730WD73ymTnE0ogSaiT/TypSoJNBF9IoEmIhplu+NNp8lIoInoE6nRRESjDGxIoImIps2aGo2k/QFm8yTMEd0ym/po9q/+T6CJ6KR6hxfUrrZAI+lc4I+q/XdTFlG7AlgA/BtwEmVlgieAXwM7AlfZ/uyIfJZSTSK066671nV4EX1teKxTr6ptUKXLGtafpiwqdjhlCZQHbR9KWUjrZMpibm+nLMB2AmUw1sh8Bm0vtL1wqnPJRMxGG4aG2tq6ocnR260Dre4G9qSss7SeMvfGBsrkOhExbW77XzfUHWh+Sam1AKymrDdN9f/qmsuKiIoNQ21u3VB3oLkFOEbSt4AHgb0lrQD2oPTXRERDOjwfzaTUetbJ9k+BN7XcdfuIJANVuoHq9nvqLD9iNuvlzuBcsBfRB7oxTcRkJNBE9AO7a2eU2pFAE9En0nSKiEYZunbquh0JNBF9olunrtuRQBPRJ9J0iojGJdBERKOcs04R0Qmp0UzRqlWrJlpovC918wMzG9/vfpAL9iKiI3J6OyIal9PbEdEo2wylMzgimpY+mohoXM46RUTjEmgiolFZezsiOiKntyOiUQY29PD57QSaiD7Ry300Ta7rFBEdNFT100y0tUPSpZLulHTGGI9vK+lGSXdIumSi/BJoIvpBm0uttFPrkXQMMNf2YmCBpD1GSfZu4ErbhwBbSlo4Xp4JNBF9YHjt7TYDzXxJK1u2pSOyGwCuqfZvBQ4epcifAL8laRtgF+AH4x1f4300kj5r+8NNlxMx203i9PZa2+PVQLYAflTtrwN2HyXNN4HfBz4EPAw8NV6BjddoEmQiOqPGPpr1wObV/jxGjxPnAO+3/ZeUQHPSeBk2HmgkLW/Z31zSDZJWSLpW0stqVJKWDlfpmj62iH4xPB9NTYFmFS82l/YD1oyS5jeAfSTNBQ6sDmFMne6j2QsYsv0GYJASLV/C9qDthRNU7SKiVY2dwcAy4N2SLgCOBVZLOmtEmnMp3+FngFcAV42XYaevo7kPeFDSzcD/oXQ0RUQN6hqCYHudpAHgCOA8248DD4xIcw+wd7t5drpGsx/wLdtvBrYFDulw+RF9aZJnnSbOz37K9jVVkJm2Ttdo1gCfkXQ68CyQfpiImszqVRBsD7TsPw0c2XSZEbOPM6gyIppll61XJdBE9InMRxMRjevl0dsJNBF9IAvIRUTzstxKRHREajQR0TRnKs+IaFoPV2gSaCL6QbmOpncjTQJND5LUtbK7/WHt5muf6br9txtPAk1EXzBDG3LWKSIalKZTRHREAk1ENC+BJiKa1sNxJoEmoi84ncER0bDhqTx7VQJNRJ9IoImIxiXQRESzbMigyohoWmo0EdEoA0Op0UREozIEISI6IRNfRUTD2l/uthumFGgkfRx4yPYySacB/w68Bdge+BfbH6jSLQfuBfa1faSkTwIP275K0ieA79q+uo4XEjHb9XKgmTPF530VOKraPxTYBXjQ9huAnSTtWz22CLjL9vAyuF8C3lHtLwGWjcxY0lJJKyVlXe6INg1PE9HO1g1TqtHYfkTSzpK2Ap4BfhNYLGkA2AbYGfjflOBzbcvzvidpyyrdg7afHSXvQWAQQFLvhuiIHuMNvft1mU4fzT3Ah4HrKU2me2xfLukPgB9UadaP8ryrgcuAE6dRdkSM0MtNp+kEmq8C3wReBQwBl0s6CVgHnDDB8z5SPTci6tDFZlE7phxobD8EvKLlrmNHSTPQelvS3sDlwDnu5XclYgbq5a9UR09v214N/F4ny4yYDTJNREQ0z+Aenvhqqqe3I6KntHdqu91aj6RLJd0p6YwJ0n1e0tET5ZdAE9EnyrU0E28TkXQMMNf2YmCBpD3GSHcIsKPtr02UZwJNRJ+YRI1m/vBFsdW2dERWA8A11f6twMEjy5K0MfAFYI2kP5zo2NJHE9EH7EkNqlxre+E4j28B/KjaXwfsPkqaPwEeAs4DTpW0q+0Lx8owNZqIPlFjH816YPNqfx6jx4nfAQZtPw5cCbxxvAwTaCL6ghkaGmpra8MqXmwu7QesGSXNo8Crq/2FwGPjZZimU0Q/qHfiq2XAHZIWUAZPHy/pLNutZ6AuBS6TdDywMfDH42WYQBPRL2qa+Mr2umrg8xHAeVXz6IERaX4GvL3dPBNoIvpAuTK4xvzsp3jxzNO0JdBE9IkMQYiIZmXt7YjohNRoIqJRGb0dEc2ruze4Zgk0EX2hT2fYi4je4t7tC06giegLpt3hBV2RQBPRB9IZHBEdkUATEQ3zZOaj6bgEmoh+UO/o7dol0ET0iwSaiGiSgaGZ3HSSNI8yXHwzyixaj1EmujkY2BpYAjwDXAu8Evge8C+2z5G0HLgX2Nf2kZI+CTxs+ypJnwC+a/vq+l9WxCwzuTmDO66dqTx3Ai6izLS1G7ADsLvtQ4GvAIcBvw38G/B64DW2z6meuwi4y/aR1e0vAe+o9pdQZvJ6CUlLh2dnn9IripiV6l3XqW7tNJ2eA04BTqKstb05JWAAPAFsQpkx/QBgBfBXLc990Pa1wzdsf0/SltXsXQ/afnZkYbYHgUEASb0boiN6zEzvDH4v8FVK8+n26r6fj0izBPiU7etG3L9+lPyuBi4DTpzEcUbEBHo50LTTdLoF+BhlISmAnUdJcz9woaRbJV0t6bXj5PdVSt/VNyd1pBExJldrb7ezdcOENRrbK4BRA4ftKwAkvQ94hNLMmgfMrx4faE0vaW/gcuAc93L4jZiBevkbVcvpbdtfoCyPOVG61cDv1VFmRLTKNBER0QEJNBHRrAxBiIimmd6+YC+BJqIvGGfiq4hoVJpOEdEJPRxnEmgi+kX6aCKiUZkzOCKalz6amambfzRJs7LsmA5nuZWIaF76aCKiWVl7OyKa1uNxpq35aCJiBqhzKk9Jl0q6U9IZE6TbQdL9E+WXQBPRD2yGNgy1tU1E0jHAXNuLgQWS9hgn+fmU6X3HlUAT0ScmUaOZP7wAQLUtHZHVAGXqXigzax48WnmSDqNM6/v4RMeWPpqIPjDJC/bW2l44zuNbUBYcAFgH7D4ygaRNgL8A3sooq5mMlEAT0SdqvPZrPS82h+YxesvnNOAi20+3c+1Vmk4RfcHVDOVtbBNbxYvNpf2ANaOkeRPwgWqRyP0l/e14GaZGE9EPDK7vwuBlwB2SFlAWjjxe0lm2XzgDZfsNw/uSlts+ZbwME2gi+kRdQxBsr6sWeTwCOM/248AD46QfmCjPBJqIPlD36G3bT/HimadpS6CJ6AcZvR0RzXNPD6qs7ayTpDMlnS3pdknfkfQqSVdVt/9O0iaS7pW0vaQfS9pJ0o11lR8x69V31ql2dZ/e3t32ocBXgBOBB6vbjwAnA/8XOBK4B3gzcN/IDCQtHb5iseZji+hrbvNfN9QdaL5U/f8E5YKeu6vbdwN7UgLLscA/AW+nnK9/CduDthdOcOViRLSwzdDQhra2bqg70Py8Zf/PgUXV/iJgNXA/8EbgFkrN5mU1moiYmjpHb9etyc7g54G9Ja0AfgicA2wN/IDShHrC9mMNlh8xq8yKs062z2zZv6La/ZsRyZ4E9qr2d66r7IiYJYEmIrqnNIsyOXlENCyBJiIal6ZTRDQugSYiGpY+mohomDOoMiI6IYEmIhpmnLW3I6JpJoEmIhqWptMM1M4SElG/bn5ZZvLfPJ3BEdEB3RuZ3Y4Emog+0a25ZtqRQBPRJ1KjiYhmdXE+4HYk0ET0AUPX5gNuRwJNRJ/IWKeIaFjOOkVEB9S19nYTEmgi+kDpC06giYhGpekUEZ2QQBMRTcvp7YhoXJpOEdGo4bW3e1UCTUSfSI1mEiQtBZZ2+zgiZpoEmkmwPQgMAkjq3XcuosfUGWgkXQrsCdxo+6xRHt8auJoSQ9YDx9n+9Vj5zantyCKiiwweam+bgKRjgLm2FwMLJO0xSrJ3AhfYPgJ4HFgyXp4dDzSS9pL0sggZEVNnw5CH2tqA+ZJWtmwjuyoGgGuq/VuBg19enj9v+5bq5nbAE+MdX8ebTrYfAs7odLkR/W4STae1theO8/gWwI+q/XXA7mMllHQQsK3tb49XYM/10UTEVNS6JO56YPNqfx5jtHwkvQK4EHjbRBmmjyaiT9hua2vDKl5sLu0HrBmZQNImlObVx2w/NlGGCTQRfaLGQLMMeLekC4BjgdWj9Ku+FzgAOF3ScknHjZdhmk4RfaDOdZ1sr5M0ABwBnGf7ceCBEWkuBi5uN88Emoi+YOz6hiDYfooXzzxNWwJNRJ/IlcER0bgEmohoWGbYi4iGZc7giOiI1GimaM999uHK66/vStkLX/2arpQLvf3L1DRJXSt7l1327FrZP/zhv04zB+MstxIRTcucwRHRuF6uCSfQRPSBOq8MbkICTURfyOntiOiArL0dEY1LH01ENKt00nT7KMaUQBPRB0xOb0dEB6QzOCIalz6aiGiYe/qs06TmDJa0g6TDplqYpLdVkxpHRI2GL9irac7g2rUdaCTtCJwPvFXSv1YTEi+XtL+kHSXdIukuSedV6XeS9HVJd0r6dJXNw8DnJW1a/0uJmN1mfKCRtAA4DzgV+Clwtu2BavsO8EngOmAxcLCkxcCfApdWy2ruL2lH26spweqiBJuIOtW3JG4TJgw0knYGzgE+aPvpMZIdCKxwCZd3Aq+jrHT3Lkk7215SzaSO7YeBz1CCzWZ1vIiIKKe32/nXDe3UaAaA+2yva7nv9Jam01xgS+Dn1WO/ALYCPgf8M3CbpI+PyPNR4FfAXiMLk7R0eE3gp37yk8m9mohZbEY3nWz/HbBO0qktd7c2nTZQ1uedVz22RXX7tcCllJXujpR0KEAVmD4LXGn7vlHKG7S90PbCbV/5yum8tohZwzZDQxva2rqhrT4a21cAayV9ZIwkdwMDKtOjvR64BzgDOMj2L4FHgM0kbQxcBHzZ9l3TPfiIeFEv12javo7G9lWSjqH011jSKdVDFwNnAlcC7wTusH2XpJ8Bg5KeB74P3AKcDlxSdSBHRI365spg29cC147x8JtGpH2Qchaq1acmU15EtK9vAk1E9LAEmohokm2Galx7u24JNBF9Ik2niGhcAk1ENCyTk0dEB2Q+mohoVNZ1iogOcE/XaCY18VVE9C57qK2tHZIureaSOmM6aYYl0ET0ibrGOlVDjeZWc0ktkLTHVNK8JH0vt+skPQk8No0s5gNrazqclJ2ymyz7Vba3m+qTJd1UHUM7NgOebbk9aHuwJa+/Bm6yfaOkPwa2tH35iPImTNOqp/topvPGA0haaXthXceTslN2L5YNYHtJjdltQZm4DsqUL7tPMc0L0nSKiJHWA5tX+/MYPU60k+YFCTQRMdIq4OBqfz9gzRTTvKCnm041GJw4ScpO2TO+7LotA+6oFiU4Cjhe0lm2zxgnzaLxMuzpzuCI6A5J2wJHUBYdeHyqaV5Im0ATEU1LH01ENC6BJiIal0ATEY1LoImIxiXQRETj/j95FtsEHH44LwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "160m 18s (- 13m 56s) (4600 92%) 0.0212\n",
      "163m 47s (- 10m 27s) (4700 94%) 0.0207\n",
      "167m 17s (- 6m 58s) (4800 96%) 0.0224\n",
      "170m 40s (- 3m 28s) (4900 98%) 0.0238\n",
      "174m 12s (- 0m 0s) (5000 100%) 0.0235\n",
      "> 玛丽 会 讲 日语 。\n",
      "= mary can speak japanese .\n",
      "< mary can speak japanese . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAD8CAYAAAAVOD3kAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAF/dJREFUeJzt3Xu4XXV95/H3JyE0kIDFCRJCqxkHtAUqEWOVGORAiYZHeaRoAUX6iNJoa/EyF4tNxqGdeGmGYZynRfBoplGhOkgrAtoKFlIuQTAHwYriZSzo0EEIgocA3nI+88dah2x21tpnJ9l7r51zPq886zl7r/07v/U9++x8z++y1m/JNhERsaNZTQcQETGskiAjImokQUZE1EiCjIiokQQZEVEjCXIPI2mupL2ajiNiJkiCbCFpL0lqOo4p/A/go00HMewkzSm/7tX+B0XSHEn57MeUNJPPg5T0cuB9FH8oZgM3Ai8BJoAFwBeAO4ATgW011cwCxmx/YgDxvhuYD+wLfN/2x/p9zJo4/gvwZdu3lInor22/UdJ1tlcM4PiPA19r2/3rwIdsX1yWuQz4C+AI4GzgIGBv4AcUv7O32f4//Y510CQdBBxh+/pd/P7XAlfb/nlvI9szzegEOUnSl22fWD5+q+2PSjoTmAv8HTAP+GXNt88Cfm57Sx/jmwOsA35k+0NlK/cDwELgPNs/6texa+JZCHwJONb2uKSrbZ/c+j72+fjftH142741FO/Px8rnzwVOBy4E/h1wBvCvwDXAk7Yf7necg1b+Xv4b8AiwApj8XLwLeAD4FMUf2Jtsv0fSwcAGYD/gRtvnSToCeDfwdts/G/CPMHTSzdjR68uvBwP/z/YjwPnApeX2auB1Lc9X9zk5HgfcAPwLcIKka4CrgaXAzcC1kl7Tr+NXxDPL9gPA8cCIpL8FjpZ0BfBbkj4n6aV9DqOuNb+tjHGO7e/b/iCwP8Xv9PUUifLPgAv6HN/ASVpE8Uf0XODHwPttj5TbnRQ/9+eAZcByScuAdwLrbS8DlkhaaPtuivfnIkm/0sgPM0xsz9gNeA5F8nkYuIKipXhZ+doFwNHl443l1+XAWuC/A0vKfV/uc4z7AouAWRWvzaJo5e41wPfsTGAT8H6KYYlZwN9Nvhflvtl9juFhYGPb9i/Am8rX1wK3UXS5J7/ntvLrqcB/bfqz1+P34xCKluD+5fPzgTe2lbkTOLJ8fAFFcjwXuAo4pKLOw4CPA3Ob/vma3Gb0bKjt+4Djy67h68ruxXGSNgLPo/hL+58AJL0TGJv8VuBMST8cQIxPSPoosEDSNuA3gHvKl2dTdCvf1O84WuK5TNJ9FOOyKmNoHX6YKPf10w9tj7TuKLvYkzGukfQPwApJv08xnvxFSe+i+IPypKRTbF/Z5zgHZQS4w/Z4y77Vks4pH/8ORTf68fL5ExQt67UUv7sbJG2w/YGW7/8e8DPgcIpx+BkpXewWtu+2/WsuuiWLKLqyk+/RXOAVLcW/C/yHAcV1lu2TgOuBmyi6+F+wfdIgk2OFPwGuA44pE9IS4O+Bt/f5uN2eaWDg88CzKYYETgFWAs/qd3KUdOSgzoiwfRkwLunclt2tXextwDjF+CMUPaVx4EhgPXAU8MpyOAdJs4EPA5fanrHJEZIgkfRs4LmSrpa0TtI/lNuNwEuBR8uioxTd3Um3A98ZUIyzyxbSgcCbgV8AR0j680Ecv4NrgfMoxrFWAnfaXmn7f/b5uHWJp/3zfAqwmiIRnFi2Ok+kaIX32/OBdwzgOADY3gBskfSemiK3UYwZC3gZxed3DXCM7ScpPstzywnBi4BP2b61/5EPtxndxZZ0FnAa8J8pWmSPtr1+OcXsHxSJ8hqK/2BA8aGU9MYBhPq7FK2yb1O00J4EfkIxOfJ121cMIIZWBwFvA/4vRfftQwM+/nPKYZCn7aOY2Z90ALAP8EHgswOKq9UCit/RwNj+tKRTKd4Ht3SxL2b7ROOZFLPYt0p6DBiV9Evg+xS9gdXAJS4mdma8nOZTQdKhFJM2Y7bfIulmisT4NYpu9Urgf1H8BX7Y9lsbinM+xSlGAz1nTdILKVpxzwOOsv3ecv9XbPd7BhtJ37H9vLZ9k7+LyfMg/xC43fZYmUy3UXS5RfG5P6HPMT7L9oP9PEb0XxLkbihPJ/lF03E0SZI85B8iSXsP+o9ITA9JkBERNWb8JE1ERJ0kyClIWtV0DO2GLabE09mwxQPDGdMwSoKc2jB+kIYtpsTT2bDFA8MZ026TdJCkmzq8PkfSNZI2SXrzVPUlQUbEtCDpAOATFCfC1zkX2Ozi+vNXS9qvY53TdZJmwYIFXrx48W7X89BDD3HggQfudj1jY2NTF4rY82yxvcv/QVauXOktW7pb62VsbOxu4Kctu0Ztj04+kbQ/xWlcn2+/FLWlzFUUK2B9U9J/pDiV74a6Y07bE8UXL17M5s2bmw7jKcO/Dm/ELrlvd755y5YtfPWrX+2q7KxZs35qe2nd65PXok/xf20ecH/5eJziooda0zZBRsSeYWKwvditFFdY/YTi2vStnQpnDDIiGmO6X3KxR8Yoli2EYpGOezsVTgsyIhpkTH9akJJOAA63/Vctuz9BsfTdsRRLud3WqY4kyIhojmHbRG8T5OQEjYv78lzf9tp9klZQtCLfVy4FVysJMiIaYwY+BontfwUu76ZsEmRENGqYTzVMgoyIRiVBRkRUsD3wLvbOSIKMiEalBRkRUcHAtiTIiIhqaUFGRNTIGGRERJXeXkbYc0mQEdGYyWuxh1USZEQ0atvERNMh1BpogpQ0BjwI/BxYCHweeAXFIpc3214taTHw/rIMts+W9AmKm5nfKumvgY/a/sogY4+IfujfYhW9MOjlzvYFfg94AfAGYAWwBngVcHJLuZMpkuDZ5fNPAm+QtDfF6hyVyVHSKkmbJW1+6KGH+vUzRESP2DDR5daEQSfIH9neSrEK8TaKFX3PAz4OtN4b4tq2JHgDcAxFIr2qrnLbo7aX2l7ai9skRET/DXg9yJ3S9BjkOHAJcBfw9Zb9T1vl1/aEpOuAC4DjBxdeRPRbJmnqvQ5YAjwEPCHpkA5lPwu81PYPBhJZRPRdE8ud7YyBJsiWhSxHyl371BR9U+sTSb8DrANW9ym0iGiCnVns3WX7H4EXNR1HRPReutgRERUMQ32aTxJkRDSqqVN4upEEGRGNShc7IqJGEmRERAVnFjsiol5akBERFXKieEREBznNJyKiRk7ziYioYJuJTNJERFTLGGQDxsbGkNR0GENt2GYP8/uamYbtc9hq2ibIiNgzJEFGRFSwnS52RESdnOYTEVHBwLYhPs8nCTIiGjXMY5CDvqthRMTTTJTjkFNt3ZC0XtImSWtqXj9A0hcl3STpkqnqS4KMiOZ0ecvXblqZkk4FZtteBiySdFhFsbOAS20fC+wnaWmnOpMgI6IxZqfui71A0uaWbVVbdSPA5eXj64HlFYd8GHi+pF8Ffh3oeJfUjEFGRKN24jSfLbY7tfjmAfeXj8eBQyvK3Ay8CngHcA/wSKcDpgUZEY3q4RjkVrbfSno+1fntA8DbbP85RYI8u1OFSZAR0ZjJ9SB7lCDH2N6tPgq4t6LMvsBvSZoNvKQMoVYSZEQ0p4eTNMCVwFmSLgROA+6WtLatzAeBUeAnwDOBT3eqMGOQEdGoXl1qaHtc0giwAlhn+wHgrrYytwNHdFtnEmRENGZyFrtn9dmPsH0me7clQUZEo3JXw4iISs5iFRERVexiG1ZJkBHRqKwHOQVJc4ENwK8BjwJvAD4DzAXus322pPOBORTnOT0DWFnOUrXWswpov/woIoZYVvOZ2irgLtvLgb8FfhO4CDgJWCzpoLLcobaPA/4GOKG9EtujtpdOcTlSRAyJHp8o3nPDkiB/A7i9fLwBeBA4B7iM4mTOycuHPll+fRDYe4DxRUQ/lLd97WZrwrAkyHuAF5eP/xTYBFwBvB54vKXc40TE9DI5UzPV1oBhSZCjwNGSNgJHA6cD76VYsgjgkIbiiog+84S72powFJM0tn9Kce1kqyPbnt/SUn5Dv2OKiMEY4jma4UiQETEzFb3n4c2QSZAR0agkyIiISmZiW67FjojYQbrYEREdJEFGRNRJgoyIqDbE+TEJMiIa5EzSRERU6vUtF3otCTIiGpUEGRFRIwkyIqKKDQ0tRNGNJMgZTFLTITzNsLUkhu39ma6G7ffeKgkyIhpjYCItyIiICrnUMCKiXlOL4XYjCTIiGuS0ICMi6iRBRkRUyHJnEREdeFsSZEREpbQgIyKqOJM0ERG1kiAjIipkubOIiDoGD/GCubOaDiAiZrJiDLKbrRuS1kvaJGnNFOU+IunkqepLgoyIRhXnQk69TUXSqcBs28uARZIOqyl3LLDQ9tVT1ZkEGRGN2okW5AJJm1u2VW1VjQCXl4+vB5a3H0vSHOBjwL2SXjNVbHvMGKSkEWDE9vkNhxIRPWLv1GIVW2wv7fD6POD+8vE4cGhFmd8HvgmsA86V9Gzbf1lXYVqQEdGoHo5BbgX2KR/Ppzq/vRAYtf0AcClwfKcK+9qClLQP8Flgf2ALReZ+EbAv8BBwBrANGAWeV+47vfz2TwGLgJ8Ar2up82CKZvQZtif/Wky+tgpob3ZHxNAyExM9m8Ueo+hWfwU4Cvh2RZnvAc8tHy8F7utUYb9bkIcDE7ZfTpEE5wM32T4O+BHwmnKbU+77AfAq4N8AXwBOpGgqH13WNx+4Cnhre3IEsD1qe+kUzfCIGBbuaQvySuAsSRcCpwF3S1rbVmY9cLykG4E/Ai7oVGG/xyDvAL4h6VrguxStwbHyta8Di8sYjpG0kSIBfgv4BfBqipbjs9jebD4HuBY4kqI1GhF7uh4tmGt7vJyrWAGsK7vRd7WVeQz4vW7r7HcL8ijgFtuvAA4AjgV+u3zthRTN3W8Dn7E9AryLIvGdCnyj/NraUrwEeDPw7nI2KiL2YMWVNL05zQfA9iO2Ly+T427rd4K8F3iHpE3AQmAz8OKytfirwNUUXeZFkv4JWEsxJnALxVjkzcAzgUPK+n5qeyvFuOYf9jn2iBiAXp4o3mt97WLbfhR45eRzSecDF9re2Fb0Dyq+/QUV+zaW9V7YmwgjolE2E0N8qeFAz4PMOYwR0S6LVUREVMhqPhERdSZnaYZUEmRENCgrikdE1PLwztEkQUZEg0wvLzXsuSTIiGhMJmkiIjpIgoyIqOSdWQ9y4JIgI6I5TgsyIqJeEmRExI4MTKSLHRFRYefuSTNwSZAR0aBcSRMRUSsJMiKiRhJkREQFG5wFcyMiqg1xAzIJMiKalEmaiIhaSZAREVVyqWFERDWTE8UjImoYZ8HciIgK6WJHRNQb4vyYBBkRzcoYZEREhdyTJiKiznQZg5S0BFhie0P/womImcXT47avtu8E7uxjLBExA02LMUhJI8AIMAr8DSDgZturJW0A9gcOAr5m+48lzQcuB+YC99k+W9L5wBxgOfAMYCUwDnwSeBbwz7bfLmkf4LNlnVuA04C928vt1k8eEc0rBiGbjqLWrF34nkOANcCrgJNb9l9h+2XAv5X0IuBg4CLgJGCxpIPKcofaPo4iyZ4ArAK+YfvlwMGSXgAcDkyU+0aB+TXlnkbSKkmbJW3ehZ8rIgZsMj92szVhVxLkL4HzgI8D+7XsHyu/fh1YDPwCOAe4DHgmsE/5+ifLrw9StAqfD/yupI3AcykS8B3ANyRdS5GEn6gp9zS2R20vtb10F36uiGiA7a62bkhaL2mTpDVTlDtI0temqm9XEuS/Bz5Ikfxao/7t8usS4HvAW4ArgNcDj7eUa30M8G3gw7ZHKFqmPwCOAm6x/QrgAODYmnIRsSezmdg20dU2FUmnArNtLwMWSTqsQ/EL2N5oq7UzCXIf4GfANcAlwFXAE5ImW3KvlnQLcI/tu4DrgPcC15ev79DiK30MOEnSjcDbgB8C9wLvkLQJWAhsrikXEXu4nWhBLpgcQiu3VW1VjVDMe0CRd5ZXHU/SCRQNtQemim3KSRpJiynGCyeAP7D9LeB/t5UB+BPb97b80DcCR7ZVd0vL6xta9p9WcehXVuyrKhcRe6idPFF8yxTDZ/OA+8vH48Ch7QUk7Q28DzgFuHKqA06ZIMukt2yKMm+aqp6IiCo9PFF8K9u7zfOp7iGfB1xk+9GyYdfRroxBRkT0SJdT2N0l0TG2d6uPohiqa3ci8PZysneJpI93qjCXGkZEcwzu3YU0VwI3SVpEcXrhGZLW2n5qRrs8TRAASRttn9OpwiTIiGhUry41tD1eXtCyAlhn+wHgrg7lR6aqMwkyIhrT69V8bD/C9pns3ZYEGRHNmS6r+URE9J6nx2IVERF9kRZkREQ1kwQZEbED20xMbGs6jFpJkBHRqEzSRETUSIKM6EI318YO0jD+xx2296gXhvF9npQEGRGNKZYymwY37YqI6IckyIiIGuliR0TUSIKMiKiUMciIiErOYhUREfWSICMiKhn3aMHcfkiCjIhGmSTIiIhK6WJHRFTIJE1ERC0nQUZE1Ml6kBERNdKCjIioUgxCNh1FrSTIiGiMyT1pIiJq5VrsiIhKmcUeGEmrgFVNxxER3ZvIpYaDYXsUGAWQNLx/liICmJyjSYKMiKgw3F3sWU0HsDMkHS5pbdNxREQPTZ7qM9XWgD2qBWn7m8CapuOIiN7JaT4RETWGuYudBBkRjbGda7EjIuqkBRkRUSMJMiKiRi8TpKT1wG8CX7S9wxkvkp4BfIYi920FTrf987r69qjTfCJiujF4orttCpJOBWbbXgYsknRYRbEzgQttrwAeAFZ2qjMtyIhojA0T3V9Js0DS5pbno+XVc5NGgMvLx9cDy4HvPv14/kjL0wOBBzsdMAkyIhq1E13sLbaXdnh9HnB/+XgcOLSuoKRjgANsf6XTAZMgI6JB7uW12FuBfcrH86kZQpT0TOAvgddOVWHGICOiUba72rowRtGtBjgKuLe9gKS9Kbrh77V931QVJkFGRKN6mCCvBM6SdCFwGnB3xdoNbwFeBKyWtFHS6Z0qTBc7IhrTy/ti2x6XNAKsANbZfgC4q63MxcDF3daZBBkRDTJ27y41tP0I22eyd1sSZEQNSU2HsINhu+qkF+/RsP1MrZIgI6JRSZAREZWGe0XxJMiIaEzuSRMR0UFakBERlYxz29eIiGq5J01ERI2MQUZEVOjllTT9kAQZEQ3KaT4REbUmMkkTEVEtY5AREVWKQcimo6iVBBkRjTE5zSciotYwT9IMbEVxSQdJOmE3vv+15XLpETGN2BNdbU0YSIKUtBC4ADhF0rfKpc43SloiaaGk6yTdKmldWf5gSV+StEnSh8pq7gE+IulXBhFzRAyCmZiY6GprQt8TpKRFwDrgXODHwPttj5TbncCfAZ8DlgHLJS0D3gmsL28AvkTSQtt3UyTZi5IkI6aHyRPFe3RPmp7ra4KUdAjwAeCPbT9aU+wlwI0u3oFNwIsp7m37RkmH2F5Z3lsC2/cAf0GRJOdWHG+VpM1tNxePiCE2YxMkMALcYXu8Zd/qli72bGA/4PHytSeA/YG/Av4euEHSn7bV+T3gZ8Dh7QezPWp76RQ3F4+IoWHwRHdbA/qaIG1fBoxLOrdld2sXexswTnGTb4B55fMjgfUU97Z9paTjAMqE+mHgUtt39DP2iBgMd/mvCX0fg7S9Adgi6T01RW4DRlTc/edlwO3AGuAY208C3wHmSpoDXAR8yvat/Y47IgZjmLvYAzkP0vanJZ1KMR5pSeeUL10MnA9cCpwJ3GT7VkmPAaOSfgl8H7gOWA1cUk7sRMQ0YJuJid7d9rXXNMwnae4OSdPzB4sZbdj+v0oa250x/9mz9/K8ec/oquxjj/14t461K3IlTUQ0atiSfqskyIhoVBJkRESdJMiIiB3ZZsLDO0mTBBkRjUoXOyKiRhJkRESl3LQrIqJW7kkTEVEh98WOiKjloW5BDuyWCxERVXp5ywVJ68s7EazZnTKTkiAjolG9Ws2nXBBndnkngkWSDtuVMq2mcxd7C3BfD+pZUNY1TIYtpsTTWc/iKVYF7IlexfSc3fz+L9le0GXZuW13Cxi1PdryfAS4vHx8PbAc+G5bHd2Uecq0TZC2D+xFPZI2D9sK5cMWU+LpbNjigeGJyfbKHlY3j+J2LVAsvH3oLpZ5SrrYETFdbAX2KR/Ppzq/dVPmKUmQETFdjFF0maG4Xcu9u1jmKdO2i91Do1MXGbhhiynxdDZs8cBwxrS7rgRuKm81fRJwhqS1ttd0KPPSThVO2xXFI2LmkXQAsILiVtIP7GqZp8omQUZEVMsYZEREjSTIiIgaSZARETWSICMiaiRBRkTU+P9OSg/bFoVMywAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Begin!\n",
    "ecs = []\n",
    "dcs = []\n",
    "eca = 0\n",
    "dca = 0\n",
    "\n",
    "while epoch < n_epochs:\n",
    "    epoch += 1\n",
    "    \n",
    "    # Get training data for this cycle\n",
    "    input_batches, input_lengths, target_batches, target_lengths = random_batch(batch_size)\n",
    "\n",
    "    # Run the train function\n",
    "    loss, ec, dc = train(\n",
    "        input_batches, input_lengths, target_batches, target_lengths,\n",
    "        encoder, decoder,\n",
    "        encoder_optimizer, decoder_optimizer, criterion\n",
    "    )\n",
    "\n",
    "    # Keep track of loss\n",
    "    print_loss_total += loss\n",
    "    plot_loss_total += loss\n",
    "    eca += ec\n",
    "    dca += dc\n",
    "\n",
    "\n",
    "    if epoch % print_every == 0:\n",
    "        print_loss_avg = print_loss_total / print_every\n",
    "        print_loss_total = 0\n",
    "        print_summary = '%s (%d %d%%) %.4f' % (time_since(start, epoch / n_epochs), epoch, epoch / n_epochs * 100, print_loss_avg)\n",
    "        print(print_summary)\n",
    "        \n",
    "    if epoch % evaluate_every == 0:\n",
    "        evaluate_randomly()\n",
    "\n",
    "    if epoch % plot_every == 0:\n",
    "        plot_loss_avg = plot_loss_total / plot_every\n",
    "        plot_losses.append(plot_loss_avg)\n",
    "        plot_loss_total = 0\n",
    "        \n",
    "        # TODO: Running average helper\n",
    "        ecs.append(eca / plot_every)\n",
    "        dcs.append(dca / plot_every)\n",
    "        ecs_win = 'encoder grad (%s)' % hostname\n",
    "        dcs_win = 'decoder grad (%s)' % hostname\n",
    "        vis.line(np.array(ecs), win=ecs_win, opts={'title': ecs_win})\n",
    "        vis.line(np.array(dcs), win=dcs_win, opts={'title': dcs_win})\n",
    "        eca = 0\n",
    "        dca = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 损失绘图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD6CAYAAABApefCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XuU5GV95/H391fXvk9P90wPwwLDZUBAFHCUi4ANgcREXU/MKp5jMCtxE6PRTfZyggejq8foLoeQzRohIZIcTkzMumcjiVHXYHAiCERBMIpCALnIDHPv+6Wu3/3jV93TU1PdXf2r+nVNV31e53Cornr69zy/+VV9+6nn93yfx9wdERFpb0GrGyAiIvFTsBcR6QAK9iIiHUDBXkSkAyjYi4h0AAV7EZEOoGAvItIBFOxFRDqAgr2ISAdItroBC4aHh33Hjh2tboaIyIby6KOPHnL3LauVO2GC/Y4dO3jkkUda3QwRkQ3FzF6op5yGcUREOoCCvYhIB1CwFxHpAAr2IiIdIHKwN7MRM3us0TIiIhK/Rnr2twJdTSgjIiIxixTszewaYAbY10gZERFZH2sO9maWBj4K3NRImUq5XzOzR8zskYMHD661KQA8tW+K3/+Hpzg8nYv0+yIinSBKz/4m4LPuPt5gGdz9Tnff5e67tmxZNQGspmcPTvOZ+57hoIK9iMiyogT7a4EPmNlu4EIz+1zEMk2RToSnkC+W46pCRGTDW/NyCe5+1cLjSjC/zcw+6e4fWa6Mu7+30YYuJ50Mg31OwV5EZFkNrY3j7qOVhx+po0wsFoK9evYiIsvb8ElVGQV7EZFVxZJUZWYDZvY1M7vXzL5UmZ0TCw3jiIisLq6kqncBt7n7dYTz7N/YQD0rWuzZlxTsRUSWE2nMfrWEKXe/fcmPW4ADUeqpRzqRADSMIyKykliSqpaUvQwYdPeHl3m94aQq3aAVEVldXElVmNlm4DPAjcuVaUZS1dFgX4r0+yIinSCWpKpK7/+LwIfdva4ts6LSDVoRkdWtOdi7+1XuPlqZP/84laSqqmK/CrwGuNnMdpvZ9Y03tTZNvRQRWV0sSVXufgdwRyPHrlcyMMw0G0dEZCUbPqnKzEgnAvXsRURWENtOVWZ2l5k9aGbLLqXQLOlkoDF7EZEVxJJUZWZvAxLufjmw3cx2NlDPqjLJQMM4IiIriGunqlHC2TgA9wFXRKmnXhrGERFZWVxJVT3AnsrjSWBkmWM1nFQFGsYREVlNXElV0xwd4uldrp5mJFVBGOyVVCUisry4dqp6lKNDN68Gno/UujplkgkN44iIrCCWnaqAe4D7zWw78PPApY02dCVp3aAVEVlRQ/PsK5m0P6oK9Lj7JOFN2oeBq919opF6VqMbtCIiK2tknv1mM7vOzIZrve7uY+7+RXdfbsZO04Rj9gr2IiLLiTr18iTgK8DrgG+a2XF3V81s0My+amb3m9kfN9jOFWk2jojIyqL27M8Hftvdfw/4OnBxjTI3AJ939yuBPjPbFbGuValnLyKyskjB3t2/4e4Pm9lVhL37h2oUOwycY2abgFOAF6M3c2WZhHr2IiIraWTM3oDrgQJQa5L7A8BO4EPAk8BYjWM0LalKs3FERJYXOdh76APAg8CbaxT5FPA+d/8EYbB/T41jNCWpKqNhHBGRFUW9Qfs7Zvbuyo+bgFrZtN3ABWaWAC4BPFoTV6cxexGRlUXt2d8J3GBm3wISwEs1dqv6dKXcBLAZ+ELkVq5CwzgiIiuLtFOVu48B11U9XZ1Y9R3CWTuxSycSlMpOqewkAluPKkVENpTYkqrWU1r70IqIrCi2pKolZW83s7dEbF9dFoJ9TitfiojUFHXD8YWkqofNbJAwqerr1YXM7Epgm7t/uYE2rko9exGRlcWWVGVmKeBPgefN7K2NNXNlmcWevYK9iEgtcSZVvRv4EXAL8Doz+2CNYzQlqWoh2GtGjohIbXEmVV0E3FlZ9fLzwNU1jtGcnaoSGsYREVlJnElVzwBnVB7vAl6IUlc9NGYvIrKyOJOq7gKurpR5P3Br9GauLK1hHBGRFcWZVDUFvD1iu9ZkYRgnV1CwFxGppb2SqkqaZy8iUst6JFWNmNljEdtXF43Zi4isLNakqopbga6I9dQlk0wAmmcvIrKcOHeqwsyuAWaAWDcdz6hnLyKyotiSqswsDXwUuGmFYzQlqao7HfbsZ3LFyMcQEWlncSZV3QR81t1rzcFfOEZTkqoGulIAjM0WIh9DRKSdxZlUdS3wATPbDVxoZp+L1sTVJRMB/dkk47P5uKoQEdnQYkuqcver3H3U3UeBx939vY01dWWDPWn17EVElhFbUlVV+dEo9azFpu40Y+rZi4jU1BZJVQCD3SnG1bMXEakptqQqMxsws6+Z2b1m9qXK7JzYDKpnLyKyrKg9+4Wkqt8jTKa6uEaZdwG3uft1hPPs3xixrrpsUs9eRGRZUcfsvwGwJKnqEzXK3L7kxy3AgSh11WuwO810rki+WF5cPkFEREJx7lS1UO4yYNDdH67xWlOSqiAcswcYn9NQjohItTiTqjCzzcBngBuXOUZTkqognI0DaChHRKSG2JKqKjdkvwh82N1j26VqwWAl2I/NqGcvIlItzp2qfhV4DXCzme02s+sbaOeqNnVryQQRkeXEuVPVHcAdEdu1Zpt7FoZx1LMXEanWNtNWFodx1LMXETlO22TQdqUTZJKBevYiIjXEui2hmd1lZg+a2bLr5jTTYHeaI7pBKyJynNgyaM3sbUDC3S8HtpvZzujNrM9Qr4K9iEgtcW5LOEo49RLgPuCK6gLNTKoCGOrNcFjBXkTkOHFm0PYAeyqPJ4GR6gLNTKoCGO5Jc3gm1/BxRETaTZwZtNNAV+VxbyN11WtzT5rD0+rZi4hUi3Nbwkc5OnTzauD5KHWtxVBvhtl8idm8Nh4XEVkqzgzaeyplbgPeQTh7J1ZDveFce/XuRUSOFWcG7aSZjVbK3eLuE5FauAbDC8F+Js8pm7vjrk5EZMOIFOzNbAD468rvTwPXu3u+qswg8JdAH3AN8L7Gmrq6oZ4MAIendZNWRGSpqMM49exCdQPweXe/Eugzs10R66qbhnFERGqLOoxTzy5Uh4FzzGwTcArwYpS61mKxZ6+59iIix2hoOuRKu1ABDwA7gQ8BTwJjjdRVj650gu50QsM4IiJVGloIjRV2oQI+BbzP3T9BGOzfU+MYTc2ghXAoRz17EZFjRZ1nX88uVN3ABWaWAC4BvLpAszNoIRzKOaSevYjIMaL27Kt3ofpYjXn2nyacjz8BbAa+EL2Z9RvuVRatiEi1qDdoV92Fyt2/Q7g65roa6snwgz2xT+kXEdlQ2manqgVDlZ69+3GjRiIiHSvqmP2AmX3NzO41sy9VxvCXK3u7mb0lehPXZqg3Q7HsTM5pfRwRkQVxJlVhZlcC29z9yxHrWbOhnoUlE3STVkRkQdTNS25393srP9ZMqjKzFPCnwPNm9tboTVyboSXr44iISCjOpKp3Az8CbgFeZ2YfbKSueml9HBGR48WZVHURcKe77wM+D1xd4xhNT6paWPnykKZfiogsijOp6hngjMrjXcBx5eJIqhrs0WJoIiLVIs2z59ikqpuBbwIpd1+6pv1dwJ+Z2TuBFPDvGmppnVKJgE3dKd2gFRFZIs6kqing7VGO36gh7UUrInKMtkuqAq2PIyJSrT2DfW+aI5p6KSKyaD0yaEfM7LHoTVw7LXMsInKsWDNoK24FuiLWE8lQT4ax2TzFUnk9qxUROWHFuS0hZnYNMEP4B2HdDPdlcA+zaEf6s+tZtYjICSm2DNrK0M5HgZtW+P2mJ1UBjPSFWbQHJnWTVkQE4s2gvQn4rLuPL3eMOJKqALYNhL35/ZPzTTumiMhGFmcG7bXAB8xsN3ChmX0uWhPXbmHoZp+CvYgIEGMGrbtftfDYzHa7+3sbaukaDPWkCQwOKNiLiAAxZtBWlR+NUk9UyUTAcG+G/RqzFxEB2jSpCsKhnP1T6tmLiEDEnr2ZDQB/Xfn9aeB6d8+vtUycRvozvDQ2t17ViYic0OJMqlpL4lXTbe3PcmBKwzgiIhBjUlW9iVdx2daf5chMnlyxRCaZWM+qRUROOHFuS7hqmbiSqiAcxgE4qN69iEisSVWrlokrqQrCYRxAM3JERIgxqarOxKvYbKsE+5cndJNWRCRqz35pUtVuM/uYmX1ylTLXN9LQtdq+KVxo8+VxTb8UEYlzW8I1JV41W382SW8myZ5x9exFRNo2qcrMOGkgq2EcERFi3qnKzO4yswfN7CO1Xo/b9k1d7NUwjohIfElVZvY2IOHulwPbzWxn9GZGEwZ79exFROLcqWqUcDYOwH3AFcDTUeqL6uRNWQ7P5JkvlMimlFglIp0rzqSqHmBP5fEkMFLj92NLqgI4aaAyI2dCQzki0tniTKqa5uhG47216oozqQqOTr/UUI6IdLo4d6p6lHDoBuDVwPNR6mrEyQr2IiJAjDtVAfcA95vZduDngUsbamkEC3vRakaOiHS6OJOqJs1sFLgOuMXdJ6LU1Yh0MmC4N629aEWk40Xt2dfF3cc4OiOnJUb6s+xXsBeRDtfIDdoRM7t/hdfPMLN/NLOHzOx3o9bTqG39WfZpNo6IdLioN2gHgbsJp1cu5zeB33X3y4CfM7PmT7epw8hAVsM4ItLxovbsS8D1hPPnl3MYONfMRoA0MB6xroYs3bFKRKRTRQr27j5Zxw3X/wdcBXyIcLZOsbpA3ElVcHRd+wPaxEREOlicq15+DPj37n4zYXLVddUF4k6qgnAYB9BQjoh0tDiD/XbgFDPLAhcDHmNdy1ro2esmrYh0sqYEezO7xsx+s+rpjwG7gYPATwkXQ1t32xb3olWwF5HO1dA8e3cfrfz/PqqCubt/BfhKI8dvhv6uJNlUoJ69iHS0tt2paoGZhXPt1bMXkQ4WW1LVknJ/Z2YXRa2nGbZv6tJetCLS0eJMqsLM3gX8xN0fi1JPs5w21MPzh2Za2QQRkZaKLamqst797wNjZnZ1xHqa4vThbsZmC0zMFlrZDBGRlokzqeq3gf8D/AnwbjP7t9UF1iOpCmDHUPgF5PnD6t2LSGeK8wbtRcBn3X0f4cqXo9UF1iOpCuD0YQV7EelscQb7Z4AzKo93AcvtaBW7UzZ3YwbPadxeRDpUnElVtwC/aWbfJlwj58+aUVcU2VSC7QNdukkrIh0rzqSqvcAvNHL8Ztox3M1zh2db3QwRkZZo+6SqBTsq0y/dW7JEj4hIS61HUtUrzewfotbTLGdt7WVirsDhmXyrmyIisu7iTqoy4DbCzUta6qytvQA8vX+6xS0REVl/ce5UBfAewo1LWm7n1j4Anjkw1eKWiIisv9iSqsxsCPhl4NYVyqxLUhXASH+GvkySZw6oZy8inSfOG7T/Hfiwuy+7RsF6JVVBuPrlmVt7eVrBXkQ6UJzB/g3A/zCz3cCFZvbJGOuqy04FexHpULElVbn72e4+WpmL/7i7f6QZdTVi50gvB6dyWhBNRDpOQ8F+aVKVu//RauVabWFGzjMHdZNWRDpLxyRVwdEZOZp+KSKdJrakKjM71cx2m9l9ZnZnZc59S528qYtsKtC4vYh0nDiTqn4d+A13vwY4BbggSl3NFATGmVt6Nf1SRDpObElV7n6zu/+48uMQcChiXU21c6uCvYh0njh3qgLAzK4Hnqisgln92rolVS3YOdLHnvE5ZnLFdalPROREEOsNWjM7A/gvwG/Ven09k6oWnLklnJHz7EH17kWkc8QW7Cvj+l8Abqz3W8B62DmiBdFEpPPEuVPVTcCpwGcqs3Le0Iy6GnXq5m5SCVPPXkQ6Spw7Vf0O8DuNHD8OqUTAaUM9CvYi0lE6KqlqwZlbenj2oPajFZHO0aHBvpfnD81QKJVb3RQRkXURZwZtysz+3sweNLMbo9YThzO39FIsOy8e0QbkItIZ4syg/SDwiLtfDrzZzPqi1BWHMysLoj2r5CoR6RBxbks4Cnyx8vhBYFd1gVYkVQGcsSX8G6VxexHpFHFm0PYAeyqPJ4GRGsdZ96QqgP5sim39WX788mpb6IqItIc4b9BOA12Vx70x17Vml5yxmQefPYy7t7opIiKxizMAPwpcUXn8auD5GOtas9efNcyh6RxP7ddGJiLS/uLMoL0b+LiZ/SFwHvDPzairWa44axiAB54+IRbjFBGJVWzbErr7C8B1wLeBa9291EhdzbZ9UxdnbOnhfgV7EekAsY6ju/ted//iibQQ2lKXnL6Z7780rnF7EWl7jSRV3VVJmPrIMq8PmtlXzex+M/vj6E2Mz7kn9TM+W2D/ZK7VTRERiVXUpKq3AYlKwtR2M9tZo9gNwOfd/Uqgz8yOm2ffaq/Y1g/Aj/dpCqaItLeoPftRjiZM3cfRWTdLHQbOMbNNhHvQvlhdoFVJVQvO2RYm9T75smbkiEh7ixrsV02YAh4AdgIfAp4ExqoLtCqpasFAV4qTN3XxpHr2ItLmogb7ehKmPgW8z90/QRjs3xOxrli9YlufMmlFpO1FDfb1JEx1AxeYWQK4BDghp7y84qQ+nj04w3zhhJoZKiLSVFGD/T3ADWZ2G/AO4Akz+2RVmU8DdwITwGbC/WhPOJecPkSp7EquEpG2FmlbQnefNLNRwqSpW9x9H/D9qjLfAc5vuIUxu/SMIfqzSb72w31ce16tWw8iIhtf5D1o3X2MozNyNqx0MuDac0f4xo/3UyiVSSVOqPXaRESaIrakqiXlbjezt0StZz288ZXbmJgr8N3njrS6KSIisYgzqQozuxLY5u5fbqCNsXv9WcMkA+OBZzRuLyLtKbakKjNLAX8KPG9mb41Yz7roySS58JRNfFvBXkTaVJxJVe8GfgTcArzOzD5YXaDVGbRLXX7WMD/YM8HEXKGl7RARiUOcSVUXAXdWZup8Hri6ukCrM2iXev2ZQ5QdHv7J4Za2Q0QkDnEmVT0DnFF5vAt4IWJd6+KiUwfpSSfY/dSBVjdFRKTp4kyqugu42sy+BbwfuDV6M+OXTgZcc+4IX39iP8VSudXNERFpqkjB3t0nCW/SPgxc7e7fd/ePVJWZcve3u/tV7n6Zu++pdawTyZsu2MaRmTzf0RRMEWkzkefZu/tYZReqfc1sUCu94eytdKUS/P0PXm51U0REmmo9kqpGzOyxqPWsp650gp87f4QvP76XmVyx1c0REWmaWJOqKm7l6MydE94Nl+1gKlfkbx474UedRETqFudOVZjZNcAMsGGGei4+dRMXnDzA3Q8+T7l8Qq7KLCKyZrElVZlZGvgocNNyBzmRkqoWmBnvvfJ0njkwzVd/qLF7EWkPcSZV3QR81t3HlzvIiZRUtdSbX7WdnVt7+YN7/5WSevci0gbiTKq6FviAme0GLjSzz0Wsa90lAuM/XXc2zx6c4QvfOW6fdBGRDSe2pKrK/PpRdx8FHnf39zbW1PX1xldu45LTN3PrPzzF2Ey+1c0REWlIbElVVeVHI7WuhcyMj7/1fKbmi/y3Lz/R6uaIiDRESVUreMW2fn7rZ3byt4/v5R5NxRSRDSy2pCozGzCzr5nZvWb2pcrsnA3n/VefxWt3DPK79/yQnx6ZbXVzREQiiTOp6l3Abe5+HeE8+zdGb2brJALjtndcCMD7//J7Wu9eRDak2JKq3P12d7+38uMWYMOuHXzK5m7+5zsv5Ml9k1z/Jw/x6AtjrW6SiMiaxLlTFQBmdhkw6O4P13jthEuqWs7PnDvC537ltRyazvNLdzzIx/72h8wXSq1ulohIXeJMqsLMNgOfAW6s9fqJmlS1nDecvYV/+q+j3Pj607n7oRf4md//J/728T24K/FKRE5ssSVVVW7IfhH4sLuf0LtUrUVPJslH33Ief/UfLmGgK8V//OvHedP/eoC7H3yeXFE9fRE5MVmUXqmZ9QP3A/8I/DzwTuDtS+fam9lvAJ8Cvl956g53/9/LHXPXrl3+yCOPrLktrVQuO3/z2B7+/NvP8cTeSXZu7eWjbzmPK84a5qn9Uzz87GEceO2OzZx7Uj+JwFrdZBFpM2b2qLvvWrVc1CEIMxsErgO+1Yy59hsx2C/1zScP8JF7fsie8TmSgVGsWlMnGRgXnzrIr1y+g589f4RUIvKsVxGRRbEH+2bb6MEeIFcs8aXv7eHFI7NsG8hy3XkjBGY88PQhnj4wzVd/8DIvHpllU3eKga4UicA4Y7iX/mySl8bnOGekj1ee3M9AV5ptA1lOGsiyf3Kes0f6yKYSLTknd8ds5W8kYzN5kgmjL5tap1aJyIL16NnfBZwLfNXdqzcbr7vMgnYI9qsplZ1vPnmArz+xj3ypTKFU5om9k8zkipyyuZun908zXWOHrHQy4ORNXWzpyzDQlSIwCMwIzDALcwECM9KJgFOHupmYK2BAJhlQLDv9XSncoVAqUyyVyZccMzhlsJueTIJ9E/OMzRZIJwNyhRKZVIJnD07z2AtjHJrJs3NrL2dt7WVyrsBgT5p8scz3XhjDgbI7+ydzZJIBV+4c5pTN3Yz0Z0kGRn82xcHpHA//5DB7x+cY7E7zmtMGOW97OKRlGIGFS1MMdKXoySQoO5w+1AMG84VS5b8y84USuWKZwe4U43MFntw3xaGpHKcNddOdTrClL0MiCCiWypw21EN/V5KJ2QIz+RJdqQSbe9KU3cmXyqQTAZlksPhHLFcscWQmT8KMTDLBfLFEsexs688yNptnNldia3+GdCJg3+Q8fdkkfdkU+WKZQ9M5DkzlODA5z8HpHGWH4Z40w30ZetJJvvfiGE/snWByrshwb5pLzxiiN5vkyEyeuXyJwZ7w3ySbSnB4OkciMIZ7M2RTCUplZzZfpDudJBEY7s7BqRz//NwRutMJTh7sIl8s89S+KXLF8uI1Ajh1qJtkYIz0ZzEzkoExOVdg78Q82zdlyRXKpBIBA10pMsmAucq/dfj+MibmCswVSgz3Zkgnj/0Wmi+W2Ts+x5a+DNO5Ii8emaVYcga6UjjO+GyBg1M5JuYKBIHx6n8zwFlbe8kkE8zmi5TKTm8mydhsgWK5TDIISCWMRGCkEgGJwBifLZArlhb/LebyJVIJI5kI6uqA5Itlyu7HXOe45Ytl5gol5vIl5golZvNF+rMpTt7URVAZwp2aL1AuQ182iRnM5Et0pxKLr0dRb7BPRjz4YlKVmd1uZjvd/em1luk0icC49rwRrj2v9kzVctn56dgs07kiLxye5cDkPFv6svzLnnH2jM1xYCq3mMVbKjtld9zDgFt2mM2XODSdW/xw5ovlmkNK6URA2f2Y5xfKZZIBuWKZ7QNZLj5tkJH+LP+6f4rvPneEge40//LSBGV3rjp7C5lkEAbn4R72Tczz4LOHeOjZw8zkj71R/Yptfezc2sfhmRx3PfDcce1plWwqIBkENf/AAov/FgtSCaNQCtseGNR7GoPdKQZ70uybmOfuh+qbq5BOBuQrdQcW/jxfKK/yW41LBoZDzaW9F2Lmeg8GdKUSzBVKmIXtK5ScwCCZCEgGttjZWdq+yfkC7uE168kkMQg7J2XHCX9Y+NyU3OlOJ+hJJ5nOFUkGRjIRHqxUDsuVyk4qEXao0smAIDByhTK54tHOyHLvazPIJhMkE8bUfPheW3pNk4HxG6Nn8p9/9pxY/x0jBXtqJ1VVB/J6ysgSQWCcNtQDwPnbBxaff9OrTqr7GLP5ItlkArPwTW8W/hFIBLb4wTAzymXn5cl55vJFhnszbOpOUy47QWCUyr7szeSFb4LL9ZbcnZl8iVLZmZwrkEkFbO3LLr4+OV/gwOR85Y8UOOEHaXy2wGy+hLvz/OEZAjOyqQRdqUT4/3RAKhFwZCZPXzbJOdv6Ge5N89LYHPOFEgemcovtf/HwLFPzBQa60/RmEszkSozN5EkkjFQQUCiXFz+ghVKZzd1hT7xUdnLFMtlUgGH85OA0W/oyDHanOTA1z1SuyCmD3UznikzPF0klArb2Z9jal2FLX4atfVnM4NB0jsPTecbnCpy/vZ8zhnswM/LFMk/snaBQcjb3pOhOJ9kzPsf3fzpOqewM9WYol52D0zkm5wp0p5N0pQMm54rkiiW60kn6s0leu2MzJXdeGpsjYcb52/tJJwOe2jfFQHc4lLZ3fI5S2dk3MY8ZFEpONpXg3wx28fL4HN2ZJMWSMzFXYL5QojudIJMMODidwzCGetNkUwkOTuXCIFa57k7Yadk+0MWBqXl6MknO3NJLMgi/DZgZm7pTDPdmGOxOMV8s89iLY7xwOOz9d6fDXuzUfIHNPWlSifDbWKHkFMtlimWnWHL6s0myqQSHpnOMzYZlc8Xw23A6EVAqO4VymVLJKbkft6vcYE+adDJgaj68VsDit8jw/QsJM4Ig/CMxmysxky/Sm0lSdqdQ9MXPZDIIv4EWyk6+WCZfLFNyJ5tMkE0FZJLh+7M7nVx8z3alA7pSCcZnC+wdn2O+8nsj/VlSifCby3yhxHBfhqn5AhefOlj3ZzyqqMG+OqnqrChlzOzXgF8DOPXUUyM2RZbqTh+9pAvxuCdz/GUOAuPkTV3HPQesOGtota/EZkZvpb6BruPH8PuzKfqbOLZ/5pZeAM5v2hEbN9Kfrfl8OhlwUdWHevumLl67Y3OkeqoDxPYl13M9gke9qt9n0hpxJlWtWmajJVWJiGxUce5UVU8ZERFZB1GHce4B7jez7VSSqszsk1UbmFSXubSxpoqISFSx7VRVo8xEY00VEZGoovbscfcxjs62iVxGRETit+ae/Wo7VFXKtMUuVSIi7WJNwb7OHaqgTXapEhFpF2sdxhmljkQpd799yY8bepcqEZF2sGKwN7M/AZbm8L4BuKvyeLlkqqW/v+wuVZXXF5OqgGkze6qeRtcwDByK+LsbWSeet865M+ic63daPYVWDPbu/utLfzazP6SOHaoqZRd2qfqlFY5/J3BnPQ1diZk9Us9CQO2mE89b59wZdM7Nt9YbtHUlSrXrLlUiIhvVWoP9PcANZnYb8A7gK2Z2nplVL1/8q8BrgJvNbLeZXd+EtoqISERrukHr7pNmNkq4Q9UtlUSpCaA6oeoO4I5mNbIODQ8FbVCdeN46586gc26yE2anKhERiY+jn/D0AAACp0lEQVQ2QhUR6QAK9iIiHWDDB/t6lm9oB2aWNLMXKze8d5vZBWb2cTP7rpn9Uavb12xmNmJm91cep8zs7yvX+cblntvoqs75ZDN7acn13lJ5vm3e77WWVal1fh1wzsd8rivlmv7Z3tDBfg3LN7SDVwFfcPdRdx8FMoTTYF8HvGRm17aycc1kZoPA3YS7nQF8EHikcp3fbGZ9yzy3YdU450uA31u43u5+sA3f79XLqryTqvPrgHO+iSWfa3f/gZntIobP9oYO9tRevqFdXQr8opk9YGZ/CVwD/F8P77B/A7iypa1rrhJwPWGWNhx7nR8Edi3z3EZWfc6XAu83s4fM7A8qz43SRu93d7/d3e+t/LgF+GWOP7/RGs9tWDXOuciSz7WZJYGriOGzvdGDffU+tyMtbEvcvgu8wd2vAMYJM5nb8tzdfbJq/4Na17mtrn2Nc/4acLm7XwacbWavos3OecHCsirAT2nz67xgyTnfy7Gf618gpnPe6MG+nr1w28W/uPvLlcdP0lnnXutc2/38H3T3qcrjJ4GdtOE5L1lW5UY65DpXnXP15zq267zR/+E6aZ/bvzCzV5tZAvhFwr/+nXLuta5zu1/7r5vZSWbWDfwc8EPa7JxrLKvS9te5xjlXf66/T0znHHmnqhNEJ+1z+wngrwAD/g74JOG5/yHhfgHtvGfA3cBXzexK4Dzgnwm/5lY/104+DnwTyAN/7O5PmdnLtNf7femyKjcDf064HMvS83Pa+5y/CfwFlc+1u3/DzALg083+bG/4DNrKLIbrgG+5+75Wt2c9mVkX8Cbge+7+k1a3J06VD/sVwNcXxrZrPdfu2v39Xuv82v2ca4njs73hg72IiKxuo4/Zi4hIHRTsRUQ6gIK9iEgHULAXEekACvYiIh3g/wM9IQcSkGdhPQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def show_plot(points):\n",
    "    plt.figure()\n",
    "    fig, ax = plt.subplots()\n",
    "    loc = ticker.MultipleLocator(base=0.2) # put ticks at regular intervals\n",
    "    ax.yaxis.set_major_locator(loc)\n",
    "    plt.plot(points)\n",
    "\n",
    "show_plot(plot_losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP0AAAEsCAYAAAAIILXCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAACipJREFUeJzt3U2IXfUdxvHn6WgSE7EmOgRn05VupDroYFUkTa0hWtxoF6GILqrMJis3NaK4KHRTxI2oMOhCSl8QoaUgZTTVYEoMNGIqXRRXWjCILxinKrUh/LqYsbGh5p5Lz/+eufN8PyDMeK/XhyHfOfeezJzrqhKAHN8YegCAySJ6IAzRA2GIHghD9EAYosdUsr3D9h7blw69ZdoQfSO2d9o+PPSOLmx/0/YfbL9k+7e2Nw296VxsXybpBUnXSXrF9uzAkzpb+3PxxpAbpiJ628/YPmL74aG3dGF7u6RnJW0bektHd0l6rKr2SHpP0q0D7xnlSkn3V9XPJC1LumbgPeN4VNIFQw5Y99HbvlPSTFXdKGnO9uVDb+rgtKR9klaGHtJFVT1ZVS+tfTor6f0h94xSVQer6qjtXVo92r829KYubN8s6TOtfmMdzLqPXtJuSc+tffyypJuGm9JNVa1U1SdD7xiX7Rskba+qo0NvGcW2tfqN9ZRWv8mua2svmR6RdGDoLdMQ/TZJ7659vCJp54BbNizbOyQ9LunHQ2/polbtl3RE0u1D7+nggKQnqurk0EOmIfpPdeY10IWajs1TZe0o9JykB6vqnaH3jGL7Adv3rH16saTBQ+rgFkn7bR+SNG/76aGGTENAr+vMU/qrJb093JQN615J10p6yPYh2/uGHjTCkqS7bb8qaUbSiwPvGamqdlXV7qraLel4Vd031Bav99+ys32RpMOS/ijpNknXT+PrZWC9WPfRS//5K7A9kl6tqkHPfALTbiqiB9CfaXhND6BHUxO97cWhN4xj2vZKbJ6E9bB3aqKXNPgXa0zTtldi8yQMvneaogfQg15P5G3y5trS6HdMTukLna/NTR67hWnbK7XbfMVVn/f+mF/64KPTmr1kpvfHfevNrb0/ptT2z8U/9PGHVTXyNw7P6/N/ukXb9B1/v8+HxAawvHx86Alj2zs3P/SEsR2s5zv9NCVP74EwRA+EIXogDNEDYYgeCEP0QBiiB8IQPRCG6IEwRA+EIXogDNEDYYgeCEP0QBiiB8IQPRCG6IEwRA+EIXogTKfobT9j+4jth1sPAtDWyOht3ylppqpulDRn+/L2swC00uVIv1ur710uSS/rzNtGS1p9xw7bx2wfO6Uvep4HoG9dot8m6d21j1ck7fzqjVW1VFULVbUwbdd5BxJ1if5TSResfXxhx/8GwDrVJeDXdeYp/dWS3m62BkBzXd7h5neSDtuek3SbpOvbTgLQ0sgjfVWtaPVk3lFJ36uqT1qPAtBOp/eyq6qPdeYMPoApxkk5IAzRA2GIHghD9EAYogfCED0QhuiBMEQPhCF6IAzRA2GIHghD9EAYogfCED0QhuiBMJ1+n76rK676XMvLx/t8yOb2zs0PPWHD42u8vnCkB8IQPRCG6IEwRA+EIXogDNEDYYgeCEP0QBiiB8IQPRCG6IEwRA+EIXogDNEDYYgeCEP0QBiiB8IQPRCG6IEwnaK3vdP24dZjALQ3Mnrb2yU9K2lb+zkAWutypD8taZ+klf91o+1F28dsH/vgo9O9jgPQv5HRV9VKVX1yjtuXqmqhqhZmL5npdx2A3nEiDwhD9EAYogfCdI6+qnY33AFgQjjSA2GIHghD9EAYogfCED0QhuiBMEQPhCF6IAzRA2GIHghD9EAYogfCED0QhuiBMEQPhCF6IMx5fT7YW29u1d65+T4fsrnlE8eHnjCWafv6Yv3hSA+EIXogDNEDYYgeCEP0QBiiB8IQPRCG6IEwRA+EIXogDNEDYYgeCEP0QBiiB8IQPRCG6IEwRA+EIXogDNEDYUZeI8/2NyX9Zu2+n0raV1X/aj0MQBtdjvR3SXqsqvZIek/SrW0nAWhp5JG+qp78yqezkt5vNwdAa50vgW37Bknbq+roWf9+UdKiJG3R1n7XAehdp+ht75D0uKQfnn1bVS1JWpKki7yjel0HoHcjX9Pb3iTpOUkPVtU77ScBaKnLibx7JV0r6SHbh2zva7wJQENdTuQ9JempCWwBMAH8cA4QhuiBMEQPhCF6IAzRA2GIHghD9EAYogfCED0QhuiBMEQPhCF6IAzRA2GIHghD9EAYogfCED0QpvPVcDeqvXPzQ08Yy/KJ40NPGNu0fY03Oo70QBiiB8IQPRCG6IEwRA+EIXogDNEDYYgeCEP0QBiiB8IQPRCG6IEwRA+EIXogDNEDYYgeCEP0QBiiB8J0it72Dtt7bF/aehCAtkZGb/sySS9Iuk7SK7Znm68C0EyXC2NeKen+qjpqe7ukayQtt50FoJWRR/qqOrgW/C6tHu1faz8LQCudLoFt25L2STol6fRZty1KWpSkLdra9z4APet0Iq9W7Zd0RNLtZ922VFULVbVwvja32AigR11O5D1g+561Ty+WdLLtJAAtdTnSL0m62/arkmYkvdh2EoCWRr6mr6qPJe2ZwBYAE8BP5AFhiB4IQ/RAGKIHwhA9EIbogTBED4QheiAM0QNhiB4IQ/RAGKIHwhA9EIbogTBED4QheiAM0QNhOl0NF+vH3rn5oSeM7cRPbhx6wtj+Of/50BPG96PnO92NIz0QhuiBMEQPhCF6IAzRA2GIHghD9EAYogfCED0QhuiBMEQPhCF6IAzRA2GIHghD9EAYogfCED0QhuiBMEQPhOkUve2dtt9oPQZAe12P9I9KuqDlEACTMfJquLZvlvSZpPe+5vZFSYuStEVbex0HoH/nPNLb3iTpEUkHvu4+VbVUVQtVtXC+Nve9D0DPRj29PyDpiao6OYkxANobFf0tkvbbPiRp3vbT7ScBaOmcr+mrateXH9s+VFX3tZ8EoKXOf09fVbsb7gAwIfxwDhCG6IEwRA+EIXogDNEDYYgeCEP0QBiiB8IQPRCG6IEwRA+EIXogDNEDYYgeCEP0QBiiB8KMvBou8P+a+/mRoSeMbfnE8aEnjG2m4/040gNhiB4IQ/RAGKIHwhA9EIbogTBED4QheiAM0QNhiB4IQ/RAGKIHwhA9EIbogTBED4QheiAM0QNhiB4IQ/RAmHNGb/s823+3fWjtn29PahiANkZdGPMqSb+uqgcmMQZAe6Oe3l8v6Q7bf7L9S9tcPReYcqOi/7Ok71bVTZJOSvrB2XewvWj7mO1jp/RFi40AejTqyP1mVX1Z8t8kXX72HapqSdKSJF3kHdXvPAB9G3Wk/4Xtq23PSLpD0l8msAlAQ6OO9D+V9CtJlvT7qjrYfhKAls4ZfVX9Vatn8AFsEPxwDhCG6IEwRA+EIXogDNEDYYgeCEP0QBiiB8IQPRCG6IEwRA+EIXogDNEDYYgeCEP0QBiiB8IQPRDGVf1dy9L2B5Le6e0B/9ulkj5s9NgtTNteic2T0HLvt6pqdtSdeo2+JdvHqmph6B1dTdteic2TsB728vQeCEP0QJhpin5p6AFjmra9EpsnYfC9U/OaHkA/pulID6AHRA+EIXogDNEDYYgeCPNvPvnSpQIKQlcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x345.6 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "output_words, attentions = evaluate(\"你 爱 音乐 吗 ？\")\n",
    "plt.matshow(attentions.numpy())\n",
    "show_plot_visdom()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 你 爱 音乐 吗 ？\n",
      "< do you love music ? <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARUAAAD9CAYAAACSjuO5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFQFJREFUeJzt3XuUXWV9xvHvQwAJCSiWFIwtXkqWAiJ0ZUTAKAOVNrhsy0JFvLWiNvVSL7VeUFJrW0ClLrQqqINRvLCwaJF6ARUXIrDA4IyKEkVAG1RqlCga493M0z/2HjIMM3POJO85Z++Z55O11+xz9nve/TLk/PLe9vvKNhERpewy6AJExPySoBIRRSWoRERRCSoRUVSCSkQUlaASEUUlqCwwkpZK2nXQ5ehE0n0GXYbYMZrP81QkrQC+RxU897a9acBFupukZbbv7NO9drX9+/r83cDHbH+6fr3I9rb6fDdgT2A/4ADgIGAl8GrbP+xxGQ8G3la/vBxYDWyrXz8UOM32Jb0swzRl2hvYz/at/bxv2833oPIZ4GXAwcBxtl80wLK8CTgC+D6wArjO9sv7dO8PA3tTBYyt9dtL6/Ov2l4raRi4ELgR+CnwM+ArwP8Co7bv6nEZjwCeAFwAvKo+fm/715LOBD5r+wu9LMOU8jwcOB/YBNxi+/R+3bvt5m3zR9KhwCbb3wSeCzxmwEV6DbCI6styLfD5ft3Y9inAPwG/B/7K9hPrSy+xvbY+3x84z/YTgLcAe9kesX1FrwNKbXzK+anAk+vX+wA/7kMZJjsEON32U4BVfb53qzW+bb0T3gt8StIq4EfAiQMuz79R/ev/z/XrU4FP9Pqmko4B1gK/oWrSfLxu5qwA3iZpd+Ac4FDg6/XHfkcVgCby2B34nftbrb3vpPL0PajY/m8ASU8GvtjPe7fdvG3+SLocWA8cCLwa2NX27QMsz8eAfwSW1D/PsL2xj/c/gaqmNNk225fX1z8P7EEVfJYCfwTcXKfbFTjF9vd7WL4h4IlUzZ9XUPWnPIaqifYI4JvAK2yv71UZpinTMuCDVLW73/brvm03n2sqbwKGbT9T0p8DdwEDCyrAp4F3U/2re1/gHVRfon45h6oJNtnbgYcA2D5W0mFUweQOqr6oVwFLbX+nT2U8kSqA/ND2SydfkPRWtnfc9ssTgA8koMzNfA4qE6Md96MKMM8YZGFsv1vSR4DLgJW2t3b6TGFLqQLFZHtOnEjaF3g/8PeTrj8AuEjS02zf2OPy7QpcCqwDzpB0GfDL+tpS4CfAD3pchqmuoGoKxhzM56Ay4b3AacCWQRai/tJ+nKpf42pJW2wP97EIP5h6P0k31T/3pqpJvdT2ekmHA9i+UdIzgUskHWG7l/0a3wJ+Yvt7wN9OKuNuVLWXC+h/UHkd1ejP6/t831abz0FlF0BUcyxureesDNJFVMHtZqqmyM2zJy/uoZKumvLecgDbWySdYvs2SScBzwe+VF/7sqQjexxQqEeYphtl2ga8AHil7fFprveyTM/v5/3mi3nbURv3JGmx7V91ke4BwB8A3+j3lzjmhwSViChq3k5+i4jBSFCJiKIWRFCRtGbQZZisaeWB5pUp5ekfSftJumaW67tJ+qSk6yQ9p1N+CyKoAE37C9G08kDzypTy9IGkfajmJy2ZJdmLqR4qPRp4oqS9ZstzoQSViJjeNuCpzD6Paxi4uD6/DhiaLcNGz1ORVGxoqmReJTStPFCmTCtXrixRFA444ACGhoaK/I7GxsZKZFPs/5lt7cznV69e7c2bN3eVdmxsbAPw60lvjdgemVSWLQDSrEVaQvXoBlTBZ7/ZEjc6qET7jI6ODroI99LhC9M6mzdv5ktf+lJXaXfZZZdf2561ZtGFrcBiqqfsJ9bhmfmeO3mziBiAcburo5Axtq8pcxiwcbbEqalEtIyBXk1alXQccLDtd0x6+/3AZZIeS7WK4qzLTySoRLSOMWWDysTDpravBK6ccu12ScdT1VZeN7Gm8UwSVCLaxrBtvL/9/Lb/j+0jQLNKUIloGUPJ/pLiElQiWqjJDwInqES0UIJKRBTjssPFxSWoRLRQaioRUYyBbQkqEVFSaioRUVST+1T68uyPpNfXG4BHxM6ycZfHIKSmEtEyvXz2p4SeBZV6RamPUO3fK+B6SRdR7TXzfeDUbCcZsWO2jTd395ReNn/WAJ+0fSzV1pErgJtsHwPcAnRc6zIipuOu/wxCL4PKQ4Cv1eejwAvZ/sj0euCg6T4kaY2kUUnNW+0nogFsGO/yGIReBpXbqdZeADgcOBc4sn59JLBhug/ZHrE9VGC1qoh5q8kdtb0MKucDT6r3790buA04RNLVVE2hC3p474h5rclBpWcdtbY3A8dOefszvbpfxEKRpQ8ioiy70aM/CSoRLbQg56lERG8YBjZc3I0ElYgWGtRwcTcSVCJaKM2fiCgqQSUiinFGfyKitNRUIqKYTH6LiOIypBwRRWVIOSKKsc14OmojoqT0qeyglStXMjranLWaJA26CI2X31F/ZPQnIopKUImIYrKXckQUlyHliCjGwLYGjyn3ZYfCiCir1Bq1ktZJuk7S2hmu7yPpMknXSHpXN2VLUIloofG6X6XTMRtJJwGLbB8NLJe0YppkzwI+ZPuxwF6SOu5ykaAS0Tbl9lIeBi6uz68EVk2T5sfAwyTdD/hj4LudMk1QiWiZib2Uuwwq+05szlcfayZltQS4oz7fAuw3ze2updpS5yXAzcBdncqXjtqIFprDkPLmWTbm2wosrs+XMn0l4yzg+ba3SHo5cCowMtsNU1OJaKESfSrAGNubPIcBG6dJsydwqKRFwKOh81h2gkpEy0ysp1IgqFwKPEvSOcDJwAZJZ0xJ8waqmsnPgPsDF3XKNM2fiLYptKVp3aQZBo4Hzra9CbhxSpobgEPmkm+CSkQLlZqmb/suto8AFZGgEtEyE6M/TZWgEtFCWU0/IgpyHiiMiHLs6miqIkFF0r8CN9u+SNK/UI13rwaWA9+nmjDzWuAq21dJejaA7QtK3D9ioWnyeiql5ql8AHhafb4a2Au4yfYxwC3Ac7rNSNKaiSnFd955Z6HiRcwvpZ5S7oUiQcX2t6meYBwGbgIOBtbXl9cDB035yGJmYHvE9pDtoWXLlpUoXsS8UnDyW0+UnFH7YeC9VLWWDcCR9ftH1q9/S1WDgao2ExE7ot6io5tjEEoGlY9SBdFrgfcAh0i6muoJxwuAjwOvrBd6+XHB+0YsPBO9tZ2OASjVUXsI8D7gLFcNud+wvY9lwk3A40rcL2Khc4OXkywSVGxvAI4okVdEdNbgwZ/MU4lom6pl09yokqAS0UIJKhFRkBnflmd/IqKQNH8iorgElYgoK0ElIkpqcExJUIloHaejNiIKynKSEVFcgkpEFJWgEhHl2DDfHyjslbGxMSQNuhh3a+K/Dk36/UT/NPHv4oRGB5WIuDcD46mpREQxmaYfEaXN+0WaIqKfBrdSfjcSVCJaKEElIorJ0gcRUZy3JahEREGpqUREOQPc0rQbCSoRLZSgEhHFNH3pg5LbnkZEPxi8bbyroxNJ6yRdJ2lth3TnSfrLboqXoBLROlWfSjfHbCSdBCyyfTSwXNKKGdI9Ftjf9ie6KV2CSkQLFdqffRi4uD6/Elg1NYGk3YDzgY2S/rqbsu1wUJF01Y5+NiJ2zhxqKvtKGp10rJmUzRLgjvp8C7DfNLf6G+AbwNnAEZJe3Kls6aiNaBl7Tg8UbrY9NMO1rcDi+nwp01cy/hQYsb1J0oeAM4G3z3bDnW7+SLqPpIskfUHShZJ2l/RaSSfW10+TdLKkPSV9VNLVks7d2ftGLGQl+lSAMbY3eQ4DNk6T5jbgofX5EHB7p0xL9Kn8HXCT7WOAW4DnAB8FTqivHwN8ClhTp3sc8ABJj5wuM0lrJqpqBcoWMQ+Z8fHxro4OLgWeJekc4GRgg6QzpqRZBxwr6WrghcCbO2VaovlzMHBJfb4eOMH2uyQ9UNLewM9s/0LSw4CjJQ0D9wMeCHxtama2R4ARAEnNHYyPGJRCDxTa3lJ/H48Hzra9CbhxSpqfA0+ZS74lgsoG4Ejgc/XPDfX7NwAvAz5ev/4WcIPt90l6IvDdAveOWJgKLdJk+y62jwAVUSKovAe4oK4efQ84q37/o8C1wIPq1+cD75N0KlVP89ML3Dtiwalm1A66FDPb4aBie7j++RvgadNc/wZw/0mvf0HVbouIndTkafoZUo5om+ylHBGlpaYSEcU0/SnlBJWItml4T22CSkTrZOW3iCjMze2nTVCJaB3TzRT8gUlQiWiZdNRGRHEJKhFRkLNBe0QUlG1PI6K4BJWIKMXAeJo/84OkQRfhXppWDW7i72jemdsatX2XoBLROplRGxGFJahERFEJKhFRjOu9lJsqQSWihRpcUUlQiWifdNRGRGEJKhFRTqbpR0RJJpPfIqIo4yzSFBHFpPkTEaU1OKYkqES0UfpUIqKYpq9Ru0uvMpa0v6TTepV/xIJV96l0cwxCz2oqtjcBb+xV/hELlxu9RcesNRVJY5Iul/Q/ktZL+qmkB9fXXi9pWNIfSvq8pGslvXvSZx8s6YJJr/eV9ElJ10t6v6Se1ZIi5juPu6tjEDp9sfcEngI8Eng68JVp0jwW+LrtVcDnZgkWrwUutH0UsAF40I4VOWKBm9hLuZtjADoFlR/a3grcDmyrf05YXP+8HFgk6QrgkfaMGzI+HLihPv8PYON0iSStkTQqabSL8kcsOCVjiqR1kq6TtLZDuv0kTVepuJe5NkGWA3tJWgQcX793FPBB28cDx0n6kxk+ezPwqPp8BHj8dIlsj9gesj00x7JFLBglOmolnQQssn00sFzSilmSv5ntFYlZzbWj9uHAfwK31QfAt4EPSNoN2MQ9azOTvQF4v6QXALcAn5vjvSMCwGa8zCJNw8DF9fmVwCrg1qmJJB0H/ILq+93RrEHF9vDkn8ABMyR93DSf3Qg8e9LrO4EndFOoiJjdHIaL953SlTBie6Q+XwLcUZ9vAQ6c+mFJuwOvA04ELu3mhpn8FtEyc5z8tnmWroStbG/SLGX67pDTgHNt/7Tb7VcyrBvRQoUmv41RNXkADmP6wZPHAy+SdBVwuKT3dMo0NZWI1ik2XHwpcI2k5cAJwCmSzrB990iQ7bu7NiRdZft5nTJNUIloG8OMEzfmko29RdIw1Uju2fUs+BtnST/cTb4JKhEtVGqavu272D4CVESCSkTLNP0p5QSViLbJym8RUdbgHhbsRoJKRBulphIRJZkElYgoxDbj49sGXYwZJahEtFA6aiOiqASV6JluH/Lql1Oe3ry1zg895tBBF+Fu554561pIXUtQiYhiqocFm7vwdYJKRAslqEREUWn+RERRCSoRUVD6VCKiIOeBwogoLUElIgoybvBeygkqES1kElQioqA0fyKimHTURkRhXe3pMzAJKhEtlPVUIqKo1FQmkbQY+C9gGfAN28/tdxkiWs3FdijsiUHspfxU4Mu2jwIeJemwAZQhorVMtUZtN38GYRDNn6uBz9Y1ln2AHwygDBGtlmd/JrH9HQBJ5wFvsf2jfpchot0y+nMvkhYBB9l+4TTX1gBr+l+qiPYotZdyLwwkqNjeJunPZrg2AowASGpuOI4YkKqftrlBZRAdtUhaBpwziHtHtJ/rdWo7H4MwqJrKncDLBnHviHkhfSoRUVK2PY2IojL6ExHFZC/liCguNZWIKKpUUJG0DjgIuMz2GdNcvy/wYapYsRV4qu3fzpbnQIaUI2LnlBhSlnQSsMj20cBySSumSfYM4BzbxwObgNWdypaaSkTrGMpMfhsGLq7PrwRWAbfe4072eZNeLgM6PlaToBLRMjaMdx9U9pU0Oun1SD1rHWAJcEd9vgU4cKZMJB0F7GP7i51umKAS0UJz6FPZbHtohmtbgcX1+VJm6A6RdH/g7cCTurlh+lQiWqfa9rSbo4MxqiYPwGHAxqkJJO1O1UR6je3buyldgkpECxV69udS4FmSzgFOBjZImjoC9FxgJXC6pKskPbVTpmn+RLRQiSFl21skDQPHA2fb3gTcOCXNO4F3ziXfBJWIlim574/tu9g+AlREgkpE6xi7udP01eTpvlmkKUpo0t/xoaEhRkdHtTN57LHHEh9wwEFdpb311rGxWUZ/eiI1lYgWalKgnCpBJaJ1svB1RBTU9DVqE1QiWig1lYgoyDhbdERESVmjNiKKSp9KRBRTckZtLySoRLROhpQjorDspRwRRaVPJSLKqTpVBl2KGSWoRLSMafaQ8pxWfpO0n6TjdvRmkp5UL08XETuh0MpvPdF1UJG0P/Bm4ERJ36yXlrtK0uGS9pd0haTrJZ1dp3+ApM9Iuk7SG+tsbgbOk3Sf8v8pEQtHoTVqe6KroCJpOXA28GLgJ8CZtofr46vAvwIfA44GVkk6GngpsK7eqOhwSfvb3kAVmM5NYInYUWZ8fLyrYxA6BhVJDwTOAv7B9k9nSPZo4GpX9a3rgEdR7SfyTEkPtL26Xv8S2zcDb6IKLHuU+I+IWEgmJr+1ufkzDHzZ9pZJ750+qfmzCNgL+EV97ZfA3sA7gMuBz0t67ZQ8bwN+Axw89WaS1kganbIBUkRM0uqgYvtCYIukF096e3LzZxvV7mZL62tL6tePANZR7SfyF5KOAaiD0FuBD9n+8jT3G7E91O8l8CLao972tJtjALrqU7F9AbBZ0qtmSLIeGJYk4DHADcBa4CjbvwJuAfaQtBtwLvBB29fvbOEjFip3+WcQup6nYvuiepf4swBLel596Z3A64EPUe0Qf43t6yX9HBiR9HvgO8AVwOnAu+rO3YjYQfPm2R/blwCXzHD58VPS3kQ1GjTZv8/lfhFxb7YZH2/uFh2ZURvRQvOmphIRzZCgEhFFJahERFkJKhFRim3GG7yXcoJKRAul+RMRRSWoRERBWfg6IgrLGrURUUzT9/2Z03KSEdEELrbym6R19eqMa3cmzWQJKhEtVCKo1A8IL6pXZ1wuacWOpJkqQSWihQot0jQMXFyfXwms2sE099D0PpXNwO0F8tm3zqspmlYeaF6ZipWnWuZnp5Uqz4MK5PEZ2/t2mXaPKasojtgeqc+XUC37CtXCagdO8/lu0txDo4OK7WUl8pE02qSV5JpWHmhemVKemdleXSirrcDi+nwp07dcuklzD2n+RCxcY2xvzhwGbNzBNPfQ6JpKRPTUpcA19RY8JwCnSDrD9tpZ0hzZKdOFUlMZ6Zykr5pWHmhemVKeHqt3yBgGvggca/vGKQFlujQ/65SvmjyJJiLaZ6HUVCKiTxJUIqKoBJWIKCpBJSKKSlCJiKISVCKiqP8HGnlOOUJqNUIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate_and_show_attention(\"你 爱 音乐 吗 ？\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "py3.6-env",
   "language": "python",
   "name": "py3.6-env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
